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About This Report 


This report presents an in-depth discussion of findings from Education Northwest’s research on 
the Alaska Statewide Mentor Project’s Urban Growth Opportunity model, which was funded 
by a federal Investing in Innovation ( i3 ) validation grant (U411B110072). The model provided 
fully released, highly skilled teachers to serve as mentors to early career teachers for their first 
two years in the profession. Using a randomized controlled trial design, researchers at 
Education Northwest examined the impact and implementation of the Urban Growth 
Opportunity as it supported early career teachers in five urban Alaska school districts. The 
purpose of the study was to examine the effects of early career teachers’ participation in the 
program on three outcomes: their retention as a teacher in Alaska, their instructional practice, 
and the academic performance of their students. 


Founded as a nonprofit corporation in 1966, Education Northwest builds capacity in schools, 
families, and communities through applied research and development. Education Northwest 
upholds professional standards in conducting and reporting research. Findings reported to the 
National Evaluation of i3 were not subject to the approval of Alaska Statewide Mentor Project, 
the University of Alaska Fairbanks (the grantee), the state, or other collaborating agencies. 
Education Northwest independently conducted all key aspects of the evaluation, including 
random assignment, collection of student assessment data, the impact analyses, implementation 
and intervention studies, and the reporting of study findings. Neither the grantee nor the 
intervention developer analyzed outcomes data for any confirmatory contrast. Neither the 
grantee nor the intervention developer reported findings (i.e., impact estimates and standard 
errors) to the National Evaluation of i3 for any confirmatory contrast. 


On the cover: View of Lake Lucille, Alaska from a mentor training site at Best Western Lake 
Lucille Inn. Photo by Phyllis Campbell Ault 
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Executive Summary 


Since 2004 the Alaska Department of Education and Early Development (AK DEED) has 
operated the Alaska Statewide Mentor Project (ASMP) as a professional development initiative 
primarily for rural early career teachers (ECTs) in the state’s high-needs districts. The initiative 
provides ECTs with two years of support from fully released, highly trained mentors. 


In 2011 the University of Alaska Fairbanks (UAF) received an Investing in Innovation (i3) grant 
from the U.S. Department of Education (DOE) to conduct a validation study of ASMP’s Urban 
Growth Opportunity (UGO) program, which adapts the ASMP model for ECTs in urban areas. 
The i3 study was designed to validate the effectiveness of the ASMP model as implemented in 
five urban areas of Alaska. This final report of the overall validation study looks at three 
interrelated aspects of UGO: implementation, intervention, and impact. 


The ASMP and UGO Mentoring Models 


The ASMP mentoring model was based on the New Teacher Center (NTC) model (Goldrick, 
2009), a well-recognized and comprehensive approach to new teacher support. Like NTC, 
ASMP uses a rigorous selection of mentors, who then participate in ongoing professional 
development, conduct weekly interactions with ECTs, collect and analyze classroom data using 
formative assessment tools, and collaborate with ECTs to develop plans for using reflective 
practices. However, ASMP adapted the NTC model to respond to the particular needs of 
teachers in Alaska, most of whom work with children from Alaska Native families that have 
unique needs. Many of these teachers also work and live in rural/bush communities in which 
they may be the only teacher in their school or one of very few. 


The ASMP model was further adapted for the UGO program to accommodate implementation 
in an urban setting (e.g., less interaction with principals, more district coordination, and greater 
flexibility to meet with ECTs more often and for shorter periods of time). The key components 
of the UGO model are based on research on teacher mentoring, including the importance of 
high-quality, experienced mentors; professional development for mentors; mentor expectations 
for interactions with ECTs; and the use of formative feedback focused on educative mentoring. 


Study Participants 


Three cohorts of ECTs participated in the study —those hired in the five partner districts in 
summer 2012, summer 2013, and summer 2014. The validation study used random assignment 
of ECTs to a treatment group (UGO participants) or a control group (non-UGO participants) 
within blocks formed by district and cohort. Teachers in the treatment group received two years 
of mentoring through UAF. Teachers in the control group either received no mentoring or 
formal mentoring as normally provided through their district. Specifically, the control group in 
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two districts participated in formal district mentoring programs, which varied in the quality 
and intensity of support provided to teachers. In three districtsno formal mentoring program 
was Offered to control teachers. The treatment group in all five partner districts received UGO 
mentoring and did not participate in any other formal mentoring offered by their district. Key 
findings from the implementation, intervention, and impact studies follow. 


Was the Project Implemented as Planned? 


The implementation study focused on how well UAF and partner districts implemented the 
project as planned. Results of the implementation study indicate that UAF implemented UGO 
with acceptable fidelity across all three years in which implementation was measured. 


The implementation study drew from an annual participant survey, which revealed several 
differences between the treatment group and control group. For example, the two groups had 
significantly different perceptions of the mentoring role. Treatment group ECTs were more 
likely than control group ECTs to think of their mentor as an expert guide, role model, 
advocate, and therapist/counselor. On the other hand, larger proportions of the control group 
considered their mentor to be a colleague, which is reasonable considering they also reported 
that their mentors were typically colleagues in their school. Treatment group ECTs met with 
their mentors less frequently but for longer periods than control group ECTs and had more 
frequent distance communication (e.g., telephone, email, text) with their mentors. Control 
group ECTs were more likely to be mentored informally and in person. As measured on the 
annual survey by a trust scale (Hoy & Tschannen-Moran, 1999), treatment group ECTs had 
higher levels of trust in their mentors than control group ECTs had in theirs. 


The treatment group and control group ECTs also had substantially different mentoring 
experiences in their first two years of teaching. This included differences in the roles mentors 
played, types of interactions they had with their mentors, and the perceived impact the 
mentoring had on their teaching practices. Treatment group ECTs reported a greater impact on 
their teaching practices than did the control group ECTs. 


What Was the Nature of the Intervention UGO Mentors and Mentees Engaged in? 


Intervention is defined as the interactions, activities, and actions mentors actually engaged in 
with their ECTs. To measure intervention, we conducted a small exploratory study with a 
sample of treatment group ECTs. This study was based on mentors’ instructional observations 
of ECTs using the Classroom Assessment Scoring System (CLASS®) collected through video 
recordings and on audio recordings of post-observation conversations between mentors and 
ECTs. Using approximately 10 percent of the 92 ECTs who had their instruction video recorded, 
we identified five ECI's who made the most gains in instructional practice (referred to as 
Gliders) and five who made the least gains (referred to as Sliders). 
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Gliders and Sliders had similar interactions with their mentors in terms of challenges, resistance 
to mentoring, and placement. However, analysis of the post-observation conversations between 
mentors and ECTs revealed intriguing differences between the two groups. For example, 
mentor-mentee pairs in the Glider group had longer conversations about instruction and 
student work, responded to each other more often, focused their conversations on how to build 
on positive practices, and engaged as peers more frequently. 


What Impact Did Participation in UGO Have on Early Career Teachers and Their 
Students? 


A major purpose of this study was to estimate the impact of ECT participation in UGO on three 
main outcomes: teacher retention in the teaching profession in Alaska, instructional practice, 
and the academic achievement of ECTs’ students in reading, writing, and mathematics. While 
no statistically significant differences were found on the confirmatory outcomes, the following 
findings emerged, suggesting promising effects: 
e Retention of treatment group ECTs in their third year of teaching was higher than 
that of control group ECTs (80.5% compared to 76.6%). While this is not a statistically 
significant difference, with an effect size of 0.16, itis a promising finding. 


e Average gains on CLASS domains of emotional support, classroom organization, and 
instructional support were higher for control group ECTs compared to treatment 
group ECTs. This is the reverse of what we would hypothesize. Differences were not 
statistically significant (effect sizes ranged from -0.32 to -0.14). 


e State assessment scores were higher for the primary students (grades 4-6) of first- 
and second-year treatment group ECTs in mathematics (effect sizes of 0.12 and 0.07, 
respectively). Assessment scores for primary students of first-year treatment group 
ECTs were higher in reading (0.07 effect size) compared to students of control group 
ECTs. In writing, the scores of primary students of second year treatment group 
ECTs were higher than the students of control group ECTs (effect size of 0.15). None 
of these differences were statistically significant. In the two subjects in which scores 
on state assessments were higher for students of control group ECTs (primary 
reading in Year 2 and primary writing in Year 1), effect sizes were both very small 
(0.04). 

e State assessment scores were higher for the secondary students (grades 7-10) of first- 
and second-year treatment group ECTs in mathematics (effect size of 0.25 and 0.17, 
respectively). Differences were not statistically significant after ECTs’ first year of 
teaching; however, the effect size of 0.25 in ECTs’ first year of teaching suggests 
substantively important differences. The effect size of 0.17 after the second year of 
UGO-mentored teaching does not rise to the level of educational significance. 


We found statistically significant differences on two exploratory analyses (at the level of p< 


0.05) suggesting important effects on specific groups of students in critical subject areas 
(secondary mathematics and primary reading): 
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e State assessment scores were higher for a diverse group of secondary students 
(grades 7-10) of first-year treatment group ECTs in mathematics. We conducted 
separate analyses of assessment scores by all reported racial/ethnic groups. The 
secondary students (grades 7-10) of first year treatment group ECTs who were identified 
as white (Caucasian), Hispanic, and Alaska Native students or students of two or more 
races (not Hispanic) obtained higher scores on the state mathematics assessment than 
students of control group teachers. These differences were statistically significant, with p 
values ranging from 0.008 to 0.037. American Indian students also achieved higher scores 
on the state mathematics assessment than American Indian students with control group 
teachers (p = 0.052). Racial/ethnic groups showing no differences between students of 
treatment group ECTs and control group ECTs were African American, Asian, and 
Native Hawaiian/Pacific Islander. 

e State reading assessment scores for students of first-year ECTs who had treatment 
mentors were higher than those of students with first-year ECTs in districts that had no 
formal mentoring programs (3 of the 5 districts). On average, the primary students 
(grades 4-6) of first-year treatment group ECTs obtained higher scores on the state 
reading assessment than students of control group teachers with no formal mentors. 
Differences were statistically significant (p =0.01) with an effect size of 0.176. 


Overall, UGO was implemented with fidelity, with room to strengthen the intervention by 
bolstering interactions between mentors and ECTs in terms of both time and activities and 
focusing on educative mentoring in which mentor-mentee partnerships consistently build on 
successes and collaboratively address instructional practice. Results of the impact study indicate 
positive effects on teacher retention and student achievement, with statistically significant 
impact on discrete groups of students. Other findings show educationally significant influence 
of the mentoring program on teachers and their students. 
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Chapter 1. Introduction 


Background 


Programs for mentoring new teachers in Alaska respond to a critical need to increase teacher 
retention, particularly in the state’s rural, remote village schools. In a 2003 speech to the Alaska 
State Legislature, Senator Lisa Murkowski pointed out that 20 percent of Alaska’s 506 public 
schools have three or fewer teachers and that the teacher turnover rate in these schools can be as 
high as 100 percent every three years (Murkowski, 2003). These turnover rates have not 
changed significantly in the subsequent 14 years since Senator Murkowski made her speech: 
Teacher turnover remains a costly and ongoing challenge in Alaska. A recent study by the 
Center for Alaska Education Policy Research calculated the total average cost of teacher 
turnover at $20,431 per teacher. Scaling that figure up to the state level—and using Alaska’s 
2008-2012 turnover data—the cost to school districts is approximately $20 million per year 
(DeFeo, Hirshberg, Cope, & Cravez, 2017). Since 2004 the Alaska Department of Education and 
Early Development (AK DEED) has largely adopted the Alaska Statewide Mentor Project 
(ASMP) as a statewide professional development initiative, predominantly supporting rural 
early career teachers (ECT), with a focus on prioritizing services for districts with the highest 
need. The project serves ECTs during their first two years in the profession by providing new 
teachers with fully released, highly trained mentors. 


The ASMP Model 

The ASMP model is based on the New Teacher Center (NTC) model, a well-recognized and 
comprehensive approach to new teacher support (Goldrick, 2009) However, ASMP targets the 
particular needs of Alaska teachers, most of whom work with children from Alaska Native 
families that have unique needs. Mentors participate in ongoing professional development, 
interact with ECTs weekly, collect and analyze classroom data, and collaborate with ECTs on 
reflective practice. AK DEED implemented ASMP in rural schools statewide; as a result, the 
program benefitted from state-level policy support, central organization, mentor training, 
logistical operation, and research. 


The UGO Model 

In 2011 the University of Alaska Fairbanks (UAF) received an Investing in Innovation (i3) grant 
from the U.S. Department of Education (DOE) to implement a validation study: the Alaska 
Statewide Mentor Project (ASMP) Urban Growth Opportunity (UGO), in five urban areasin 
Alaska. The UGO study aimed to validate the effectiveness of the ASMP in an urban setting. 
The rigorous study included implementation, intervention, and impact studies, which are 
summarized in this report. The implementation and intervention studies used mixed-methods 
designs; the impact study used a randomized controlled trial. In summer 2012, participating 
urban school districts hired the first of three cohorts of mentors who then received training to 
begin working with the first cohort of newly hired ECTs. Each summer for three years (2012, 
2013, and 2014), we randomly assigned all of these ECTs to either treatment or control 
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conditions. ECTs assigned to the treatment (UGO) condition received a UGO mentor for two 
years and no other district- or school-level formal mentoring. Those assigned to the control 
condition received their district’s regular program (business-as-usual or BAU) for ECTs. Two of 
these districts used formal mentoring programs. No formal mentor supports were available to 
ECTs in the three remaining UGO districts. 


Each year researchers engaged in data collection activities to support the three studies. 
Education Northwest researchers participated in many of the UGO training events, and 
collected data from UAF and mentors regarding mentor participation in professional 
development activities and interactions with ECTs. We arranged for video recording of ECTs’ 
instruction, administered surveys, and conducted interviews. We also collected teacher 
retention and student achievement data from AK DEED. 


Literature Review 


In the past 20 years there has been a dramatic increase in the number of teachers who are just 
entering the profession in the U.S. This is referred to as the “greening” of the teacher force by 
Ingersoll, Merrill, & Stuckey (2014), who report that in 1987-88 the typical teacher had 15 years 
of teaching experience. By 2011-12 the typical teacher was in their fifth year of teaching. A 
recent analysis of data from the U.S. Department of Education’s Office for Civil Rights showed 
that nationally, 12 percent of all public school teachers are in their first or second year of 
teaching. In some states, the percentage is closer to 15 percent (Sawchuk & Rebora, 2016). 


Recently, the National Commission on Teaching and America’s Future (NCTAF, 2016) used 
“research-based evidence” to update its recommendations for improving the U.S. educational 
system. The commission recommended several strategies to improve teaching and learning. 
One of the six key strategies described is mandatory, state-supported, multiyear induction and 
mentoring for new teachers. This recommendation reflects the growing body of evidence on the 
effectiveness of mentoring. In a critical review of 15 empirical studies, Ingersoll and Strong 
found that induction for beginning teachers—and teacher-mentoring programs in particular — 
have a positive impact on teacher dispositions (Ingersoll & Strong, 2011). Nearly all the studies 
they reviewed “showed that beginning teachers who participated in some kind of induction 
had higher satisfaction, commitment, or retention” (Ingersoll & Strong, 2011, p. 38). 


Key Components of Mentoring 

Although mentoring programs are common in the United States (Strong, 2009), they can vary 
widely in quality, quantity, and the types of activities in which mentors and mentees engage. In 
their review of the literature Hobson, Ashby, Malderez, and Tomlinson (2008) reported a 
several findings directly related to the ASMP mentoring model, which are further described in 
the following sections. 


Quality of mentors. Hobson and colleagues (2008) found that selecting mentors and matching 
them with ECTs is a critical feature of the mentoring relationship. The quality of the mentor’s 
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teaching experience is important, as is their ability to impart knowledge, provide support, 
listen, and respond to needs. Likewise, the extent to which ECTs see and acknowledge that 
expertise via “professional respect” is crucial. Developing this professional respect is tied, in 
part, to the extent to which the ECT and mentor share common teaching assignments. 


Mentor training/professional development. It is important for mentors to be trained 
appropriately for the role (Hobson et al., 2008). The most effective training includes seminars 
that involve groups of mentors and educators and opportunities for outside conversation to 
address isolation and engage in conversations that build skills (Hobson et al., 2008). Effective 
mentor training is research based and addresses how to interact personally with ECTs and how 
to conduct reflective practice (Hobson et al., 2008). 


Expectations for mentor interactions with ECTs. In their review of the literature, Hobson and 
colleagues (2008) also found that trust is essential for a successful mentoring relationship 
because it encourages ECTs to share personal and professional challenges. Trust typically 
develops as ECTs see that their mentor listens and responds to their needs and as the mentor 
shows their commitment to the relationship by engaging regularly with the ECT in both formal 
and informal ways. In an extensive review of the literature, Hoy and Tschannen-Moran (1999) 
found that trust is a multidimensional construct and a critical element to learning. Through 
their analysis, Hoy and Tschannen-Moran concluded that trust relies on one party's willingness 
to be vulnerable to another. This willingness to be vulnerable is based on the confidence that the 
other is benevolent, reliable, competent, honest, and open (Hoy & Tschannen-Moran, 1999). 
Tschannen-Moran reports that “compelling evidence is accumulating on the importance of trust 
to high-performing schools” (Tschannen-Moran, 2017). 


Another critical component of successful mentoring relationships is dyad dynamics. In a study 
of successful and failed mentoring relationships in academic health centers, Strauss and 
collaborators (Strauss et al., 2013) identified five essential features of successful mentoring 
relationships: reciprocity, mutual respect, clear expectations, personal connections, and shared 
values. Adult learning theory is central to this idea, and we know froma large body of literature 
that adult learners need to be involved in the planning and evaluation of their instruction, that 
experience — including mistakes—should be the basis for learning activities, that adults are most 
interested in learning about subjects that have immediate relevance to their job or personal life, 
and that adult learning should be problem-centered rather than content-oriented (Knowles, 
1984; Smith, 2002; Merriam, 2001). Even though ECTs are not entrenched in their teaching 
practices, effective educative mentoring often relies on changing their behavior. The literature 
on changing adult behaviors suggests that adult learners have preferences for active learning 
strategies that support cognitive growth, transformational learning, immediate application of 
knowledge, and opportunities for self-direction (Ross-Gordon, 2011). 


Use of formative feedback for educative mentoring. Relationship-building is another 


cornerstone of effective mentoring. Mentors and ECTs benefit from taking time at the beginning 
of the relationship to get to know each other personally and professionally (Hobson et al., 2009). 
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This includes assessing strengths and areas of growth and setting goals. An important feature of 
goal setting is to ensure that the ECT has influence and autonomy over the direction of 
mentoring and the development of their “teaching style.” The ability fora mentor to provide 
feedback that allows the ECT to grow hinges on their ability to observe ECTs’ practice. An 
observation cycle that includes a pre-observation conference, observation, and a post 
observation conference that is “(i) conducted in a sensitive, nonthreatening way; (ii) focuses on 
specific aspects of mentees’ teaching; and (iii) provides an opportunity for genuine and 
constructive dialogue between mentor and mentee which includes joint exploration of the 
perceived strengths and weaknesses of the mentee’s teaching, discussion of the likely impacts of 
observed teaching actions, and the development of ideas which might help the mentee 
overcome any problems or weaknesses” (Hobson et al., 2009, p. 212). Educative mentoring is a 
specific type of mentoring that focuses on improving mentees’ practice (Achinstein, & 
Athanases, 2005; Feiman-Nemser, 2001). Educative mentors find openings, pinpoint problems, 
probe new teachers’ thinking, notice signs of growth, and focus on students. These 
characteristics are reinforced by the literature on the need to engage mentees in activities that 
will actively engage their learning. A critical piece of observation cycles is engaging mentors 
and ECTs in sustained “professional dialogue” focused on developing ECTs’ instructional skills 
(Lofthouse, Leat, Towler, Hall, & Cummings, 2010). 


Organizational support. Evidence is growing that teacher mentoring has greater benefits when 
it is part of a schoolwide support system for the induction of new teachers. Ingersoll (2001) 
noted that organizational factors within a school, such as lack of support from principals, 
student discipline issues, and lack of input and decision-making power, cause teachers to leave 
the profession. In their 2006 review of empirical literature on recruitment and retention, 
Guarino, Santibanez, and Daley reported that inservice school policies “that provided 
mentoring and induction programs, particularly those related to collegial support, had lower 
rates of turnover among beginning teachers.” Their synthesis of the research also revealed that 
“[s]chools that provided teachers with more autonomy and administrative support had lower 
levels of teacher attrition and migration” (Guarino et al., 2006, p. 201). In their meta-analysis of 
teacher career trajectories, based on findings from 34 studies of 63 attrition moderators, Borman 
and Dowling (2008) examined attrition and factors that moderate attrition. Their analysis found 
“that initiatives that lessen the bureaucratic organization of schools and school systems and 
strategies that promote more genuine administrative support from school leaders and 
collegiality among teachers are strategies that may improve retention” (Borman & Dowling, 
2008, p. 399). 


Implementation/intervention science. Implementation science seeks to investigate “what is 
actually enacted, how an innovation is enacted, and why the contexts, conditions, characteristics, 
and other influences shape innovation enactment as they do” (Century & Cassata, 2016, p. 172). 
Implementation and intervention are sometimes referred to as part of a “black box” that 
evaluation research has historically struggled to open. Adding to the confusion is that different 
researchers use different labels to describe the implementation/intervention distinction. Some 
researchers use the terms implementation and intervention interchangeably (e.g., Greene, 2015), 
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while others contrast “structural dimensions of fidelity” to “process dimensions of fidelity” 
(Harn, Damico, & Stoolmiller, 2017) or simply compare implementation to “innovation” 
(Century & Cassata, 2016). In our approach, “implementation” is the processes or methods by 
which the intervention is adopted and put into use, while “intervention” is the process or 
methods used to actually promote improvements or changes in outcomes (Dunst, Trivette, & 
Raab, 2013). Implementation, then, is the training, resources, and requirements for mentors, all 
of which are intended to promote the delivery of high-quality mentoring to novice teachers (the 
intervention). Researchers in implementation science contend that implementation and 
intervention should be considered as distinct yet overlapping aspects of a program because 
effective implementation (e.g., training to use an intervention such as mentoring) does not 
guarantee an effective intervention (e.g., high-quality mentoring in practice) (Fixsen, Naoom, 
Blase, Friedman, & Wallace, 2005; Dunst et al., 2013). 


Key Impacts of Mentoring 

The impact of mentoring on supporting and retaining excellent teachers has important 
implications for teacher preparation programs, state departments of education, and school 
districts. Several impacts are associated with ECT's who work with a mentor teacher. These 
include increased retention in the teaching field and, to a less-documented extent, improved 
instructional practice and increased student achievement. 


Retention. The largest body of research on teacher mentoring and collegial support is associated 
with teacher retention. In an analysis of national survey data, Smith and Ingersoll (2004) used 
data from the 1990-2000 Schools and Staffing Survey and its Teacher Follow-up Survey. They 
found that, among the 3,000 beginning teachers in the survey sample, those who participated in 
induction and mentoring programs in their first year of teaching were less likely to leave 
teaching or change schools. In addition, Smith and Ingersoll found that the more types of 
support teachers experienced, the lower the probability that they would change schools or leave 
the teaching profession. In their study, on average, 29 percent of beginning teachers either 
changed schools (15%) or left teaching (14%). The types of induction support that had the 
strongest positive association with retention were having a mentor in the same field, having 
common planning time with other teachers in the same subject, having regularly scheduled 
collaboration with other teachers, and being part of an external network of teachers. In another 
study, Ingersoll and Kralik (2004) reviewed 150 empirical studies on induction and mentoring 
programs. The researchers found 10 studies that could be included in their analysis because 
they met their criteria (used quantitative data; well-defined, verifiable outcomes; and a 
comparison group). The review did not provide definitive evidence of the value of mentoring 
programs in keeping new teachers in the profession; however, the authors reported that 
“collectively the studies do provide empirical support for the claim that assistance for new 
teachers and, in particular, mentoring programs have a positive impact on teachers and their 
retention” (Ingersoll & Kralik, 2004, p. 14). Ingersoll and Kralik also concluded that 
“conspicuous by their absence in this field are careful, randomized or quasi-randomized 
experimental studies involving random-assignment procedures and controlled trials with a no- 
treatment control group. This kind of approach is perhaps the most expensive, but also is 
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potentially the most fruitful” (Ingersoll & Kralik, 2004, p. 15). Ina separate research review 
conducted for the Education Commission of the States, researchers agreed that the empirical 
evidence linking mentoring to teacher retention was significant but limited (Allen, 2005). 


A 2006 review of empirical literature on teacher recruitment and retention examined the 
characteristics and policies of schools that show evidence of successfully recruiting and 
retaining teachers (Guarino et al., 2006). The review concluded that “[s]chools that provided 
mentoring and induction programs, particularly those related to collegial support, had lower 
rates of turnover among beginning teachers” (p. 201). Ina recent study, the Regional 
Educational Laboratory (REL) Central, in collaboration with Aurora Public Schools (CO),used a 
randomized controlled trial to examine the impacts of a mentoring program on student 
achievement, teacher retention, and teacher evaluation ratings (DeCesare, McClelland, & 
Randel, 2017). In the study, the district’s elementary school teachers were randomly assigned to 
receive either the district’s typical mentoring support only (the business-as-usual/BAU group) 
or to receive both BAU mentoring support and added support from a retired mentor. 
Researchers found that “although more teachers in the program group than teachers in the 
business-as-usual group left the district after two years, the effect of the program on teacher 
retention was not significant” (p. i). The study also found that the amount of time ECTs were 
mentored was associated with higher retention rates during their second year of participation. 
“Mentees who received more hours of mentoring were more likely to stay in the district. The 
odds of a mentee staying in the district doubled with each additional 10 hours of mentoring. 
The sharpest increase in retention occurred with each additional hour received after 25 hours” 


(pi). 


Instructional practice. A review of the benefits of ECT participation in mentoring found limited 
evidence of impact on teaching practice, but did find some evidence that ECTs had experienced 
impact on their “behavior and classroom management skills and ability to manage their time 
and workloads” (Hobson et al., 2008). In a recent review of 32 empirical studies with teacher 
coaching (comparable in definition to ASMP mentoring) as an outcome, Kraft, Blazar, and 
Hogan (2017) found large positive effects on instructional practice. 


Student achievement. There is very little empirical research that connects improved student 
achievement to mentoring. In their recent study of a mentoring program using retired teachers 
to mentor elementary ECTs, DeCesare, McClelland, and Randel found somewhat mixed results 
(DeCesare et al., 2017). Student gains in mathematics achievement were statistically 
significantly higher in the students of teachers in the program group after the first year of 
mentoring. At the end of the second year, scores of students of teachers in the program were 
higher (effect size of 0.06) but not at a statistically significant level. Reading achievement was 
higher among students taught by formally mentored teachers in the program group than 
students of teachers in the control group (effect size of 0.07), but the differences were not 
statistically significant (DeCesare et al., 2017). In their meta-analysis of empirical studies on the 
impact of mentoring (referred to as “coaching” by these researchers) Kraft, Blazar, and Hogan 
(2017) found positive effects on student achievement. The largest effects were in programs that 
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focused on teachers’ instructional practice in reading. The pooled effect from these studies was 
.18 standard deviations on students’ reading achievement (Kraft et al., 2017). 


UGO Logic Model 


Evidence from the mentoring literature on quality mentors, professional development, 
interactions with ECTs, and using formative feedback is reflected in key components of UGO, 
including the importance of high-quality, experienced mentors; professional development for 
mentors; mentor expectations for interactions with ECTs; and the use of formative feedback 
focused on educative mentoring. To map out a clear picture of UGO, we created a logic model 
of the program (figure 1). Logic models provide a simplified picture of programs, including 
goals, plans for meeting those goals, and a description of what program success would look like 
(Kekahio, Cicchinelli, Lawton, & Brandon, 2014; W. K. Kellogg Foundation, 2004). Together, 
Education Northwest and UAF identified four key components of the program: selection and 
assignment of highly qualified mentors; in-depth professional development for mentors; 
structured interactions between mentors and ECTs; and use of formative assessment tools. 


Selection and Assignment of Highly Qualified Mentors 

Each participating UGO district hired a district coordinator to act as a liaison between the 
district, mentors, UAF, and the external research team. To ensure that districts identified and 
hired highly qualified mentors to work with ECTs, UAF staff members collaborated with 
districts, using a structured set of application questions and interview rubrics to identify 
potential mentors. The ASMP model includes a minimum qualification for becoming a mentor 
of at least eight years of teaching experience in Alaska. In assessing the quality of potential 
mentors, the ASMP model also stipulates that candidates have recent classroom experience or 
other relevant work in the education field (within two years), have strong content knowledge in 
core subjects, be recognized as an excellent classroom practitioner, and demonstrate 
commitment to improving academic achievement for all students. 


Once hired, mentors are fully released from classroom responsibilities and dedicate their full- 


time equivalent (FTE) to mentoring. To provide sufficient time for each mentor to spend with 
assigned ECTs, a full-time mentor has a caseload of no more than 15 ECTs. 
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Figure 1. UGO logic model 
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In-Depth Professional Development for Mentors 

Mentors receive ongoing and in-depth professional development. Beginning in the summer 
prior to mentoring, all mentors attend orientation that introduces new mentors to the project 
and reviews program changes for continuing mentors. Following orientation, mentors 
participate in four trainings (approximately one-weeklong) per year. UAF gears some of this 
training content tonew mentors and other content to all mentors. Typically, the first three days 
of training cover the New Teacher Center (NTC) curriculum. Certified NTC trainers in Alaska 
provide this required training to first- and second-year mentors. The specific focus is on the 
ASMP model of mentoring, which is an adaptation of the NTC model (referred to as Academy). 
The remaining training involves all mentors and includes ASMP-specific training and the 
Friday Forum. ASMP training is devoted to topics specifically related to implementing the 
ASMP model, and Friday Forum provides an opportunity for mentors to address relevant 
issues and spend time preparing materials and networking. Sometime during the week, the 
ASMP also provides a Cultural Connections training. 


Between these intensive, weeklong trainings, mentors participate every other Friday in a virtual 
Friday Forum. These three-hour meetings address successes, challenges, and other pertinent 
issues that arise through mentors’ work. Friday Forum is ongoing mentor professional 
development and networking time used to keep the mentors connected to each other and the 
program and to support them in the field. They also receive ongoing support from a coach who 
communicates with them twice monthly and shadows them during their face-to-face visits with 
ECTs. “Shadowing” involves coaches joining mentors for a site visit, participating in classroom 
observations and debriefs alongside mentors, and offering feedback to mentors post-ECT 
interaction. Coaches shadow first-year mentors twice, and once per year thereafter. During 
communications and shadowing, coaches provide feedback on mentors’ use of ASMP materials 
and strategies using the Mentor Accountability and Growth Assessment System (MAGA). 
Coaches are all highly experienced mentors who have completed the NTC and ASMP training 
multiple times. 


A year-end Wrap Up session takes place each May. During this time, mentors review all the 
documentation completed during the year for each ECT and submit it to the ASMP research 
team. 


The ASMP model, while derived from the NTC model (both Academy trainings and Friday 
Forums), includes modifications to fit the needs of Alaska, such as restructured time with ECTs 
(while maintaining minimum criteria); restructured timing of NTC’s formative assessment 
system to fit the Alaska calendar year; and a focus on working with students of diverse cultural 
backgrounds, especially Alaska Native students. 


During the four years in which UAF implemented UGO, the state experienced reduced 
revenues caused by changes in international oil prices. (The oil industry has long beena 
primary employer and revenue source for the state.) State agencies across the board accepted 
budget cuts, including UAF and its Office of Academic Affairs & Research, K-12 Outreach, 


Mentoring ECTs in Urban Alaska 9 


which houses the ASMP. In response, UAF altered some of the training by reducing the number 
of days mentors participated, moving the in-person meetings from Fairbanks to the more 
centrally located Anchorage and requiring mentors who lived within driving distance of the 
city to commute daily. While these changes condensed meeting time, they did not change the 
content of the professional development. 


Structured Interactions between Mentors and ECTs 

The training ultimately prepares mentors to interact with their ECTs in a way that involves both 
trust and reflection. ASMP expects mentors to maintain weekly contact with each ECT, usually 
through phone, email, or Skype. In addition, ASMP expects at least monthly in-person contact 
between mentors and each ECT, totaling at least three and a half hours per month. All 
conversations should focus on instructional practices (educative mentoring). On-site visits 
provide an opportunity for mentors to collect data in an area of the ECT’s interest, review and 
analyze the data, and suggest instructional strategies or develop a plan to move ECT practice 
forward. Mentors base their work on responsiveness to ECT needs, which includes district- 
specific needs, school-specific needs, Alaska-specific needs, and culturally responsive needs for 
the region. Mentors document all of their contacts using a variety of ASMP tools. 


Mentor Use of the Formative Assessment Tools 

During training, mentors learn about, and gain experience using, the project's formative 
assessment tools. The tools provide a structured means for documenting work with each ECT. 
For example, mentors use a Collaborative Assessment Logs (CALs) to capture ongoing 
communications between them and their ECTs. Other tools mentors use on site with their ECTs 
include classroom observation and other instructional data collection (e.g., seating charts, I 
Notice I Wonder, or Selective Scripting tools). Mentors use some tools to promote reflective 
practice —the Individual Learning Plan (ILP), Mid-Year Gauge (MYG), and Professional Growth 
Reflection (PGR). Mentors and ECTs review a self-assessment and complete an ILP in fall/early 
winter. Together they establish areas of growth for the ECT, and they revisit these through the 
MYG. They complete a PGR in the spring to allow ECTs to reflect on their progress and growth 
during the year. All documentation includes areas for mentors and ECTs to identify relevant 
Alaska Teaching Standards, including Alaska Standards for Culturally Responsive Schools. 


Overview of the Study 


The study used random assignment of ECTs to treatment (UGO) or control (business as usual, 
or BAU) groups at the teacher level, within blocks formed by district and cohort. Teachers in the 
UGO group received two years of mentoring through UAF. BAU teachers were eligible to 
receive any formal district mentoring, orno mentoring, as normally provided through the 
district’s BAU. The BAU condition was district-specific, varied in quality and intensity, and 
included the absence of mentoring. UGO-mentored ECTs received UGO mentoring and did not 
receive any other formal mentoring offered in their district. Three cohorts of ECTs 

participated —those hired in partner districts in summer 2012, summer 2013, and summer 2014. 
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RCT Design 

Within each of the three cohorts, teachers in each district were randomly assigned to UGO or 
BAU conditions from the beginning of each academic year through October 1 of that year, based 
on time of hire. The probability of assignment to UGO and comparison groups was equivalent 
across districts and batches. Teachers were identified for randomization if they met the 
eligibility criteria for ECTs hired within the specified time windows. 


Education Northwest received lists of ECTs hired across districts from either UAF or each 
district coordinator; lists were received in batches to accommodate hires made after the first day 
of school. We combined district lists and sorted the list in alphabetical order by the ECT’s first 
and last name. ECTs were assigned a random number via a random-number generator. We then 
re-sorted the list by district and ascending random number. Within each district, teachers on the 
first half of the list were assigned to the treatment group (UGO) and the second half were 
assigned to the control group (BAU). In the case of an odd number of ECTs, for the first batch of 
randomization we assigned the middle ECT to the treatment group, the second batch the 
middle ECT was assigned to the control group, and so on. In the case of only one ECT being 
hired in a district, we used a coin toss, with heads indicating assignment to the control 
condition and tails indicating assignment to the treatment condition. In each ECT’s first year of 
teaching, for purposes of analysis, students were randomly assigned to experimental conditions 
because they were attached to ECTs at the time ECTs were randomly assigned. 


Research Questions 
Under the broader aim of validating the ASMP model in an urban environment, we conducted 
three studies: implementation, intervention, and impact. 


The purpose of the implementation study was to examine UGOas it was put in place by UAF. 
The research questions for the UGO implementation study were: 


1. What was the overall level of fidelity of implementation? 


2. To what extent were the UGOkey components implemented as planned — mentor 
recruitment/assignment, training, contact, and use of formative assessment tools? 


How much variation in implementation was there across mentors? 
In what ways were key components implemented differently from the model as planned? 


5. What were the facilitating conditions and challenges to implementation? 


The purpose of the intervention study was to examine what UGO mentors actually did as they 
interacted with their ECTs. We framed the intervention study with this research question: What 
patterns in UGO mentor-ECT conversations are associated with improved ECTs’ instructional 
practice? 


The purpose of the impact study was to estimate the effect of ECT participation in UGO on key 
outcomes. Research questions for the impact study were: 
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1. What is the impact of ECTs’ participation in UGO on remaining in the teaching 
profession in Alaska? 
What is the impact of ECTs’ participation in UGO on their instructional practices? 
What is the impact of ECTs’ participation in UGO on the reading, writing, and 
mathematics achievement of their students? 


Samples 
Implementation study. We included all mentors and their ECTs in all five districts. Some 
indicators included in the UGO key components involved fewer mentors, as appropriate. 


Intervention study. We included UGO ECTs with scores on the CLASS, drawing a subsample of 
ECTs with the greatest and least gains. This subsample included approximately 10 percent of 
ECTs with instructional practice outcome data. 


Impact study. We included all ECTs randomly assigned to UGO and BAU conditions in the 
retention analyses. 


The instructional practice analysis included a randomly selected subsample of ECTs from the 
full sample of Cohort 2 and Cohort 3 ECTs randomly assigned to the UGO and BAU conditions. 
We used a replacement process due to varying district requirements as to which teachers could 
be video recorded. A priori, across all districts, we excluded preschool teachers and teachers 
providing instruction in languages other than English and Spanish. At Anchorage, we excluded 
special education teachers and teachers assigned to schools on military bases, per district 
requirements. We excluded Cohort 1 ECTs from the study because in the first year we were still 
negotiating video recording logistics with districts. 


The student achievement analysis included, from the original full sample of randomly assigned 
ECTs, those who taught reading, writing, and mathematics to students in grades 4-6 and ECTs 
who taught mathematics to students grades 7-10. As Alaska administers the state assessment to 
students in grades 3-10, we excluded ECTs who taught in grade 3 because their students would 
not have baseline scores and those who taught grades higher than grade 10 because their 
students would not have outcome scores. Since most primary classrooms are self-contained 
and/or students receive the bulk of their instruction from their homeroom teacher, we included 
all eligible ECTs teaching grades 4-6. In the secondary grades, teachers vary, and language arts 
is often integrated into instructional practice across the curriculum, making it difficult to 
attribute reading and writing assessment scores toa single ECT. Therefore, we only included 
secondary ECTs who provided mathematics instruction. 


Data Sources and Outcome Measures 

We used data from multiple participants, collected by multiple instruments, and administered 
by different entities for the various studies. Participants included administrators, trainers, 
coaches, district coordinators, mentors, ECTs, and students. Instruments included interview 
protocols, applications, rosters, records, audio and video recordings, surveys, and assessments. 
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Data were collected by Education Northwest researchers and contractors, UAF administrators, 
mentors, coaches, and AK DEED. Each of these is displayed below (Table 1) and further 
described in the text that follows. 


Table 1. Data sources 


Differences between 
conditions 


Implementation 


Data source 


|Administratorteveldata data |Administratorteveldata Ft 


[aretinrearrecrennpenrarenvenrvsccmaen| [x] 
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Interviews. We used interview data in the implementation study and to describe differences 
between the BAU and UGO conditions. Each spring we conducted interviews with protocols 
aligned to the four key components of the intervention to ascertain how different stakeholders 
experienced the intervention and encountered successes and challenges. We interviewed UAF 
staff members (including administrators, professional development providers, and coaches); 
district coordinators; district mentoring program administrators; and mentors. In most cases, 
we conducted these interviews in person; otherwise, we conducted them via telephone. 


Forms, surveys, and other documentation. Data collected on forms, surveys, and other extant 
documents were used for the implementation study. Both Education Northwest and UAF 
collected these data. UAF shared their data with Education Northwest. To document mentor 
full-time equivalence (FTE) and the extent to which mentors were fully released to serve as 
mentors, UAF shared their annual mentor application with us. We augmented these data with a 
brief annual mentor profile form we collected each fall. 


Professional development participation data were documented and collected in a variety of 
ways. In all years, we collected our own mentor attendance data while observing many of these 
events. For any in-person training event we did not attend, we either collected attendance data 
from UAF and/or the district coordinators. For Friday Forum, we received a participation roster 
froma Friday Forum co-facilitator who tracked this. 


We used multiple data sources to document mentor interactions with ECTs. These included 
mentor’s weekly contact with ECTs documented in the Contact Log by Calendar Week form. 
This documents the means by which mentors contacted ECTs (phone, email, face-to-face visits) 
and total minutes spent in face-to-face activities. Another source of mentor level data is the 
folder mentors maintain for each ECT. These folders hold CALs that mentors complete weekly, 
summarizing successes, challenges, next steps for both the mentor and ECTs, and standards 
addressed during conversations. They also contain the various formative assessment tools 
mentors used with ECTs, including classroom observation tools and reflective practice tools. 


Coaches also collected data from their interactions with mentors in a folder. These documents 
included their bimonthly communication with mentors and associated CALs, coaching 
activities, and reflective practice tools. 


We administered an annual ECT survey to both BAU and UGO ECTs. In survey analyses, we 
combined all UGO ECT surveys across all cohorts and years and compared results to all BAU 
ECT surveys combined across all cohorts and years. In total, we analyzed 1,049 surveys, 512 
from BAU ECTs and 537 from UGO ECTs. A large section of the survey included items 
applicable only to ECTs who had mentors. In these analyses, we considered an ECT as having a 
mentor if they had a UGO mentor, a formal mentor (a district-assigned mentor in Anchorage 
and Kenai or a school-assigned mentor), or an informal mentor (either in their school or outside 
their school). 
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Audio recordings. Mentors audio recorded selected post-observation conversations between 
themselves and ECTs as part of the intervention study. During their second year of mentoring, 
we asked mentors to record four conversations with each of their second-year ECTs—two 
recordings each semester. These were to be post-observation debrief conversations. To ensure 
the recordings reflected the diversity and breadth of the work mentors did, we requested that 
mentors space recordings for a given ECT at least one month apart and that mentors capture a 
variety of different types of conversations. Mentors could choose when to record these 
conversations and ECTs could choose not to have certain conversations (or portions of 
conversations) recorded. Mentors used audio recorders and uploaded the audio files to a secure 
upload site. 


Instructional practice data. We used observations of instructional practice for the intervention 
and impact studies. We obtained data on instructional practice through video recordings of 
ECTs conducted by video technicians hired by Education Northwest. Cohort 2 and Cohort 3 
ECTs randomly assigned to the instructional practice study were video recorded three times in 
the fall of their first year of teaching, three times in the spring of their first year of teaching, and 
three times in the spring of their second year of teaching. The three recordings at each time 
period were made within a given recording window of about one month. 


Instructional observation recordings were scored by independent raters trained in using the 
Classroom Assessment Scoring System (CLASS®) (Pianta & Hamre, 2008). We selected this 
instrument for several reasons: (1) psychometric properties have been calculated and described 
in the literature, (2) the scales were found to be reliable and predictive of student gains in 
another recent professional development study (Allen, Pianta, Gregory, Mikami, & Lun, 2011), 
(3) all the domains are observable through a recorded classroom observation, and (4) a 
preliminary alignment study showed that CLASS was aligned with the ASMP Professional 
Teaching Standards and the Standards for Alaska’s Teachers. 


We provided video recordings to certified CLASS raters who scored the ECTs’ instruction, blind 
to ECTs’ UGO or BAU condition and the time period in which the recordings were made. 
Raters employed the CLASS elementary, upper elementary, and secondary protocols. 
Depending on the length of each recorded instructional period, raters scored “cycles” of about 
15 minutes of instruction in each of 10 dimensions across three domains —emotional support, 
classroom organization, and instructional support. Raters double-coded 3 percent of the video 
recordings. We calculated interrater reliability to ensure consistent scoring of the observations 
across raters. Across the three protocols we obtained interrater reliability of 89 percent. We 
obtained interrater reliability of 84 percent for the elementary recordings, 92 percent for the 
upper elementary recordings, and 91 percent for the secondary recordings. 


CLASS uses a 7-point scale, with scores of 1 and 2 in the low range; 3, 4, and 5 in the midrange; 
and 6 and 7 in the high range. For each observation, we averaged up to three cycle scores for 
each dimension and then averaged the dimension scores to calculate domain-level scores. We 
then averaged the domain-level scores across up to three observations from each time period. 
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Most ECTs had scores from the same observation protocol in both Years 1 and 2, but some ECT 
who changed positions between their first and second years of teaching had scores from 
different protocols. We analyzed the domain scores for emotional support, classroom 
organization, and instructional support from all ECTs pooled across the three protocols. In 
addition, as analyses indicated differences in results across the elementary and secondary 
protocols, we conducted analyses of these scores for the elementary scores only and for the 
upper elementary and secondary scores combined. 


Teacher retention data. We used state-level teacher employment records for the analysis of 
impact on teacher retention. We received these data directly from AK DEED under a data- 
sharing agreement. AK DEED maintains a database of teachers currently teaching in the state 
and provided files that documented the Alaska public schools each ECT taught in from 2012-13 
through 2016-17. If a teacher is included in the file for a given year it indicates that they are 
teaching in the state; their absence from the records indicates they are not teaching in the state. 
This enabled us to calculate retention for Cohort 1 ECTs in their third, fourth, and fifth years of 
teaching and for Cohorts 2 and 3 ECTs into their third year of teaching. 


Student achievement data. We used student achievement data from state-administered 
assessments for the analysis of impact on student achievement. We collected information from 
district coordinators regarding the classroom rosters of ECTs teaching reading, writing, or 
mathematics to grades 4-6 students and ECTs teaching mathematics to grades 7-10 students. 
For these students we received state assessment data from AK DEED. Data from 2011-12 were 
baseline data for students from the spring administration of the reading, writing, and 
mathematics Standards Based Assessment (SBA). We also received reading, writing, and 
mathematics SBA data from students assessed in spring 2013 and spring 2014. The spring 2015 
data we received in reading, writing, and mathematics were from the pilot administration of the 
Alaska Measures of Progress (AMP). We received “claim-level “data from the English language 
arts assessment to calculate separate reading and writing scores. Alaska cancelled the spring 
2016 administration of the AMP due to technical difficulties. We calculated z-scores for each 
student on each assessment using statewide assessment means and standard deviations for each 
subject, grade, and year, and we used these z-scores in our analyses. Additional details 
concerning the standardization are given in the discussion of analytic methods. 


Setting and participants. UGO was implemented in five urban districts across Alaska. The 
number of coaches, mentors, ECTs, and students was driven by the number of newly hired 
teachers within each district. The following sections describe the districts and participants. 


Urban Districts 

In 2011, UAF collected data on Alaska school districts and spoke with district administrators to 
recruit districts to participate in the UGO study. Ultimately, they selected districts to participate 
if they were in an urbanized area of Alaska, agreed to participate in the study, and anticipated 
new teacher hires. Five of Alaska’s largest urban districts participated— Anchorage, Fairbanks 
North Star Borough (Fairbanks), Kenai Peninsula Borough (Kenai), Matanuska-Susitna Borough 
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(Mat-Su), and Sitka. Nonew districts were added after 2011. Table 2 displays select 


demographics for the five districts during the 2010-11 school year. 


Table 2. UGO district demographics 


Anchorage Fairbanks 


Total preK—12 enrollment 49,206 
Total schools 98 
Total teachers (in FTE) 2,973 


Enrollment by 
race/ethnicity 


American 

Indian/Alaska Native BO2e 
Asian 5,128 
Hispanic 5,030 
Black 3,183 
White 23,250 
Hawaiian/Pacific 2,052 
Islander 

Two or more Races 6,231 

Enrollment of other student groups 

ELL/LEP 5,351 
SPED 6,964 


14,285 
35 
844 


1,569 


335 
1,018 
877 
9,071 


121 
1,294 


384 
2,235 


Kenai 
9,327 
44 
594 


1,131 


126 
283 
60 
7,308 


48 
371 


201 
1,329 


Mat-Su Sitka Total 
17,079 1,388 91,285 
44 6 227 

993 99 5,503 
1,971 413 9,416 
300 116 6,005 
389 43 6,763 
264 10 4,394 
13,664 732 54,025 
118 8 2,347 
373 66 8,335 
419 46 6,401 
2,569 179 13,276 


Source: CCD 2010-11 Version 2a. (U.S. Department of Education, National Center for Education Statistics, 2012) 


Mentors 


Beginning in summer 2012, districts hired the first cohort of UGO mentors. Over the next two 
years, districts hired new mentors to replace those who did not return and to ensure ideal 
caseloads as additional ECTs were hired. In the last year of the study, districts did not hire any 
new mentors. Table 3 displays mentor participation by district and year. 
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Table 3. UGO mentors 


Year Mentors Anchorage Fairbanks Kenai Mat-Su Sitka Total 
New 4 2 1 2 1 10 
2012 
Total 4 2 1 2 1 10 
New 4 3 1 1 0 9 
2013 Returning 3 2 1 2 1 10 
Total 7 5 2 3 1 18 
New 1 0 1 0 0 2 
2014 Returning 6 4 1 3 1 15 
Total 7 4 2 3 1 17 
Returning 4 3 1 2 1 11 
2015 
Total 4 3 1 2 1 11 


Of the 21 mentors who served in the study, the majority were female (85.7%). Their average age 
was 50.5 years. They spent 29.6 of these in Alaska and taught for 21.7 of them. 


Early Career Teachers 

The overall study sample came from the population of newly hired ECTs in the five 
participating districts. We defined ECTs as district-contracted teachers in their first year in the 
teaching profession, hired after October 1 of the previous school year and before September 30 
of the current school year, and assigned to a position that included the development of lesson 
plans and a student list/roster. We randomly assigned new hires to UGO and BAU conditions in 
2012-13 (Cohort 1), 2013-14 (Cohort 2), and 2014-15 (Cohort 3). Once hired, ECTs participated in 
the study for two years (Cohort 1 through 2013-14, Cohort 2 through 2014-15, and Cohort 3 
through 2015-16). Each year at least one UGO ECT stopped working with their mentor (n = 13). 
This was typically because the ECT was put ona plan of improvement (54%); otherwise ECTs 
dropped their mentor because they felt they were ready to “be on their own” or “had enough 
support in their building.” One UGO ECT declined to work with their UGO mentor in order to 
work witha district mentor more aligned with his/her subject area assignment. Table 4 shows 
the number of ECTs included in each study by cohort and year. 


Table 4. ECTs included in studies, by cohort and group 


Retention and 


implementation Instructional practice Student achievement Intervention"? 
Cohort UGO BAU UGO BAU UGO BAU UGO 
1 83 73 NA NA 25 15 NA 
2 105 102 31 30 32 25 — 
3 98 95 39 34 22 25 — 
Total 286 270 70 64 79 65 10 


‘No BAU teachers were included in theintervention study. 
2Data are notreported to protectthis group with 10 or fewer teachers. 


Table 5 provides additional detail regarding ECTs in their first year of teaching. 
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Table 5. Select characteristics of ECTs at baseline 


Characteristics 


No mentor 
Formal mentor 
Informal mentor 
Female 


White 


30 years or 
younger 


31—40 years 

41 years or older 
Bachelors 
Master's 


Degree in AK 


Did not relocate to 
AK 


1 Cox index. 


Anchorage 
UGO BAU 
7% 3.7% 
(1) (5) 
98.6% 94.9% 
(144) (129) 
0.7% 1.5% 
(1) (2) 
76.7% 81.6% 
(112) (111) 
85.6% 77.3% 
(125) (102) 
56.8% 59.6% 
(83) (81) 
28.1% 24.3% 
(41) (33) 
15.1% 16.2% 
(22) (22) 
59.6% 57.4% 
(87) (78) 
24.0% 27.9% 
(35) (38) 
64.2% 67.2% 
(88) (86) 
83.9% 86.3% 
(115) (113) 


Fairbanks 
UGO BAU 
0.0% 61.1% 

(0) (22) 
100.0% 2.8% 
(32) (1) 
0.0% 36.1% 
(0) (13) 
83.9% 77.8% 
(26) (28) 
81.3% 80.6% 
(26) (29) 
46.9% 57.1% 
(15) (20) 
37.5% 25.7% 
(12) (9) 
15.6% 17.1% 
(5) (6) 
65.6% 72.2% 
(21) (26) 
21.9% 22.2% 
(7) (8) 
80.6% 77.1% 
(25) (27) 
90.3% 97.1% 
(28) (34) 


Kenai 
UGO BAU 
0.0% 24.1% 

(0) (7) 
100.0% 62.1% 
(37) (18) 
0.0% 13.8% 
(0) (4) 
72.2% 72.4% 
(26) (21) 
97.1% 93.1% 
(34) (27) 
45.7% 55.2% 
(16) (16) 
34.3% 31.0% 
(12) (9) 
20.0% 13.8% 
(7) (4) 
77.1% 58.6% 
(27) (17) 
20.0% 20.7% 
(7) (6) 
37.5% 37.0% 
(12) (10) 
60.6% 57.1% 
(20) (16) 


Mat-Su 
UGO BAU 
0.0% 44.6% 

(0) (25) 
100.0% 12.5% 
(58) (7) 
0.0% 42.9% 
(0) (24) 
74.1% 75.0% 
(43) (42) 
92.6% 85.7% 
(50) (48) 
36.2% 41.1% 
(21) (23) 
37.9% 33.9% 
(22) (19) 
25.9% 25.0% 
(15) (14) 
58.6% 69.6% 
(34) (39) 
29.3% 19.6% 
(17) (11) 
65.4% 56.6% 
(34) (30) 
96.5% 90.7% 
(55) (49) 


Sitka 
UGO BAU 
0.0% 11.1% 

(0) (1) 
100% 22.2% 
(8) (2) 
0.0% 66.7% 
(0) (6) 
75.0% 66.7% 
(6) (6) 
87.5% 88.9% 
(7) (8) 
62.5% 55.6% 
(5) (5) 
37.5% 444% 
(3) (4) 
0.0% 0.0% 
(0) (0) 
50.0% 44.4% 
(4) (4) 
50.0% 55.6% 
(4) (5) 
62.5% 62.5% 
(5) (5) 
37.5% 57.1% 
(3) (4) 


UGO 


0.4% 
(1) 
99.3% 
(279) 
0.4% 
(1) 
76.3% 
(213) 
88.0% 
(242) 
50.2% 
(140) 
32.3% 
(90) 
17.6% 
(49) 
62.0% 
(173) 
25.1% 
(70) 
63.1% 
(164) 


83.1% 
(221) 


All 


BAU 
22.6% 
(60) 
59.0% 
(157) 
18.4% 
(49) 
78.2% 
(208) 
81.7% 
(214) 
54.7% 
(145) 
27.9% 
(74) 
17.4% 
(46) 
61.7% 
(164) 
25.6% 
(68) 
62.9% 
(158) 


84.7% 
(216) 


Effect size’ 
-2.664 


2.768 
-2.510 
-0.064 
0.301 
-0.110 
0.125 
0.009 
0.009 


0.015 


0.003 


-0.073 


Mentoring ECTs in Urban Alaska 19 


In most cases, UGO and BAU ECTsshared similar demographics. Most were female, of similar 
ages, and had earned similar credentials. Similar proportions of ECTs also earned their degree 
in Alaska and did not relocate to the state for their position. While the racial/ethnic background 
of both UGO and BAU ECTs groups was predominately white (80%), a statistically significant 
larger number of UGO ECTs identified themselves as white compared to BAU ECTs (88% to 
82%, respectively). Finally, and as expected, UGO ECT’ were statistically significantly more 
likely to have a formal mentor, and BAU ECTs were statistically significantly more likely to 
have no mentor or an informal mentor (as would be expected). 


Students 
The student sample consisted of students assigned to participating ECTs (as described above). 
District coordinators collected classroom rosters, as of October 1 each year, from eligible ECTs. 
In collaboration with the ECT, district coordinators “cleaned” the rosters by: 
1. Removing any student who received instruction in the subject area from more than one 
teacher (e.g., students who had another teacher for reading or mathematics) 


2. Removing any student who did not take the regular state assessment (e.g., special 
education students) 


3. Adding any student who was enrolled as of October 1 but did not appear on the roster 
and/or removing any student who was not enrolled as of October 1 but who did appear 
on the roster (ie., roster error) 


These students comprised the denominator for calculating attrition. 


At the time of randomization (first year of ECT teaching), ECTs and their students were 
randomly assigned to UGO or BAU groups. 


In both years, the majority of students were white (at least half); Alaskan Natives were the next 
largest group of students. Few students were African American, American Indian, or Native 
Hawaiian/Pacific Islander. About half of students were female. At one point in the past three 
years, the majority had been eligible for free or reduced-price lunch [FRL] (about two-thirds), 
were not limited English proficient [LEP] (about four-fifths), or had not received special 
education services [SPED] (about four-fifths). Table 6 describes the demographics of the 
student samples included in the four confirmatory analyses. 
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Table 6. Demographic characteristics of students, Year 1 
Secondary math Y1 


Category 
Race/Ethnicity! 
African American 
Alaskan Native 
American Indian 
Asian 
Hispanic 
Native Hawaiiar/ 
Pacific Islander 


Two or more 
races 


White 
Female 
Male 
FRL 
Not FRL 
LEP 
Not LEP 
SPED 
Not SPED 


Primary reading Y 1 


BAU 


5% 
10% 
<5% 
10% 
10% 


5% 


10% 


55% 
49.7% (286 
50.3% (290 
67.0% (386 
33.0% (190 

16.7% (96 
83.3% (480 
18.4% (106 


) 
) 
) 
) 
) 
) 
) 
81.6% (470) 


UGO 


5% 
15% 
<5% 
10% 

5% 


5% 


10% 


55% 
48.8% (287) 
51.2% (301) 
61.2% (360) 
38.8% (228) 
13.6% (80) 
86.4% (508) 
20.2% (119) 
79.8% (469) 


81.2% (491 


Primary writing Y1 


BAU UGO 
5% 5% 
10% 15% 
<5% <5% 
10% 10% 
10% 5% 

5% 5% 
10% 10% 
50% 50% 

49.8% (301) 47.6% (429 

50.2% (304) 52.4% (472 

68.4% (414) 68.6% (618 

31.6% (191) 31.4% (283 

17.9% (108) 18.1% (163 

82.1% (497) 81.9% (738 

18.8% (114) 19.2% (173 

) 


80.8% (728) 


Primary math Y 1 
BAU UGO 
5% 5% 
10% 10% 
<5% <5% 
10% 5% 
10% 5% 
5% 5% 
10% 5% 
55% 60% 


48.4% (281 
51.6% (299 
67.1% (389 
32.9% (191 

16.7% (97 
83.3% (483 
20.2% (117 
79.8% (463 


KS YH lm wma lwo os | oo_—soC 


1 Percentages rounded to the closest 5 percentage points and numbers removed to protect individuals. 


51.7% (307 
48.3% (287 
62.6% (372 
37.4% (222 

13.8% (82 
86.2% (512 
20.0% (119 
80.0% (475 


~ Holo wooo SS wes WH 


BAU 


5% 
10% 
<5% 
10% 
10% 


5% 


10% 


50% 
46.7% (596 
53.3% (680 
71.8% (916 
28.2% (360 
21.2% (270 

78.8% (1006 
28.4% (362 
71.6% (914 


~~ S# VS JH Oar OU aS TC 


UGO 


5% 
10% 
<5% 
10% 
10% 


5% 


5% 
50% 
49.3% (668 
50.7% (687 
62.7% (849 
37.3% (506 
17.7% (240 
82.3% (1115 
19.3% (261 
80.7% (1094 


~~ wom DB meoi /wees wa 
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Table 7. Demographic characteristics of students, Year 2 
Secondary math Y2 


Category 
Race/Ethnicity! 


African 
American 
Alaskan Native 
American Indian 
Asian 

Hispanic 

Native Hawaiiar/ 
Pacific Islander 


Two or more 
races 


White 
Female 
Male 
FRL 
Not FRL 
LEP 
Not LEP 
SPED 
Not SPED 


Primary reading Y2 


BAU 


5% 
15% 
<5% 
10% 
10% 


5% 


10% 
45% 
46.7% (202) 
53.3% (2 
69.5% (3 
30.5% (132 

19.6% (85 
80.4% (348 
20.8% (90 
79.2% (343) 


31 
01 


) 
) 
) 
) 
) 
) 


UGO 


76.5% (372 
19.5% (95 
80.5% (391) 


) 
) 
) 
29.4% 143) 
) 
) 
) 


Primary writing Y2 


BAU 


<5% 
10% 
10% 


5% 


10% 
45% 
47.1% (210) 
52.9% (236) 
70.6% (315) 
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5% 
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47.4% (184 
52.6% (204 
76.5% (297 
23.5% (91 
22.2% (86 
77.8% (302 
21.9% (85 
78.1% (303 


WN Sara ese oo BS wa YS |e YH 


' Percentages rounded to the closest 5 percentage points and numbers removed to protectindividuals. 
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UGO 


BAU 


UGO 


Multiyear Participation 
The study actively involved participants across four school years and used data collected over 
six school years. Table 8 displays participation and data collection over the six-year period. 


Table 8. Study participation and data collection 
2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 


1 ECT Cohort 1 Y1 Y2 
2 ECTCohort2 Y1 Y2 
3  ECTCohort3 Y1 Y2 
4 Retention, fall C1 C2 C3 
5 CLASS (baseline, 
fall Y1) C1 C2 C3 
6 CLASS (outcome C1 C2 C3 
spring Y1) 
7 CLASS (outcome, C1 C2 C3 
spring Y2) 
a en C1 C2 C3 
baseline) 
9 SBA(Y1 C1 C2 C3 
outcome) 
10 SBA (Y2 C1 C2 C3 
baseline) 
11 SBA(Y2 , 
outcome) C1 C2 C3 
12 Mentor group Y1 New Returning Returning Returning 
13 Mentor group Y2 New Returning Returning 
14 Mentor group Y3 New Returning 


Notes: Rows 1-3 showECTs participated for the duration oftwo schoolyears; during that time they remained in the 
condition to which they were assigned. Row 4 shows we assessed retention for each cohort at the beginning of their 
third year of teaching. Rows 5-7 showwe collected CLASS data for the instructional practice study in the fall 
(baseline) and spring of ECTs’ first year of teaching and the spring of their second year of teaching (outcomes in Year 
1 and Year 2). Rows 8-11 showwe collected data for the student achievement study for the first four years. The year 
prior to randomassignment of cohort 1 ECTs was baseline data for the students assigned to Cohort 1 ECTs in their 
firstyear of teaching; outcome data was from the end of their first year of teaching. This was the same for Cohorts 2 
and 3. Similarly, the year of random assignmentof Cohort 1 ECTs was baseline data for the students assigned to 
Cohort 1 ECTs in their second year of teaching; outcome data was collected at the end of their second year of 
teaching. This was the same for Cohorts 2 and 3. Each year, school administrators assigned ECTs anew class of 
students. The state administered assessments annually in the spring of 2013, 2014, and 2015 (no state tests were 
administered in 2016). Rows 12-14 show districts hired new mentors the first three years ofthe study. For the 
implementation study, we only assessed years of experience the first year mentors were hired, and we only assessed 
participation in the NTC training the first two years of mentoring. All other measures included all mentors all years. 


1 Alaska did not administer a state assessment in spring 2016. The students of eligible ECTs were not 
included in Year 2 analyses. 
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Chapter 2. Differences between the UGO and BAU 
Conditions 


Research Question 


An important component of any rigorous study is identifying the different conditions into 
which participants are placed. In our study this translated into a research question that is 
anchored in implementation, and could influence impact, specifically: How did the UGO and 
BAU conditions differ? 


Methods 


To determine significant differences between UGO and BAU conditions we drew from 
interview and survey data. Each year researchers interviewed district coordinators and 
administrators of district-sponsored mentoring programs to learn about the regular supports 
available to ECTs in their districts, and specifically to ECTs assigned to the BAU condition. 
Using ethnographic techniques, we analyzed the content of interview responses and identified 
similarities and differences between mentoring opportunities available to UGO and BAU ECTs. 


We also analyzed data from the annual ECT survey to determine areas of ECT experience in 
their first years of teaching with statistically significant differences between responses from 
UGO and BAU ECTs. Survey analyses combined all UGO ECT surveys across all cohorts and 
years and compared results to all BAU ECT surveys combined across all cohorts and years. On 
survey items in which ECTs were asked to select practices or topics and their impact, we 
analyzed responses using chi-squared distributions. For the trust scale we used a t test of 
significance. A large section of the survey included items applicable only to ECTs who had 
mentors. We used some of these questions to further examine what distinguished the 
experience of ECTs with UGO mentors as distinct from BAU mentoring (those who reported 
having a formal district-assigned or school-assigned mentor or an informal mentor either in 
their school or outside of their school). 


Interview Findings 


District Mentoring Support (BAU) 

All districts provided ECTs with some support. Under BAU conditions, content coaches were 
common. Content coaches were provided to ECTs in Kenai's non-Title I schools; language arts 
and mathematics ECTs in Fairbanks; and special education ECTs in Mat-Su, Fairbanks, and 
Kenai. ECTs in Anchorage received targeted support from various colleagues based on teaching 
assignments. These included teacher experts, curriculum specialists, or content coaches. Sitka 
administrators paired ECTs with a veteran teacher to help them learn about district policies and 
procedures. Importantly, district-supported instructional mentoring was available to ECTs in 
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Anchorage and those in Title I schools in Kenai. The district coordinator reported that the 
design of the Anchorage mentoring program was influenced by the ASMP model. This was 
confirmed by annual interviews with the BAU coordinator in Anchorage. As only Anchorage 
and Kenai had formal mentoring programs, the following sections compare these BAU 
mentoring programs to UGO mentoring. 


Mentor Recruitment 


There was one similarity across district mentoring programs related to mentor recruitment — 
both the UGO and Kenai mentoring programs used full-release mentors; otherwise, the three 
programs had different mentor experience and caseload requirements. UAF expects UGO 
mentors to have at least eight years of teaching experience in Alaska. Anchorage mentors were 
required to have at least five years of teaching experience (this was reduced to four years over 
the course of the study), with three of them required to be in Anchorage. The Kenai BAU 
program did not have minimum requirements established for mentors in regard to their time 
teaching in the district or in Alaska. 


Unlike UGO’s full-release mentors and caseloads capped at 15 ECTs, Anchorage mentors were 
full-time teachers who were afforded a day of shared sub time to work with their mentees 
during the school day. To make it easier for mentors and ECTs to meet, beginning in Year 3, 
Anchorage administrators tried matching mentors with ECTs in the same building. Anchorage 
mentors usually had one mentee. Kenai used fully released mentors. In Years 1 and 2, the Title I 
mentors did not have a maximum caseload; in Year 3, the district capped caseloads at 14 ECTs. 


Professional Development 


The professional development required by UAF for UGO mentors was substantially more 
comprehensive than that provided by Anchorage or Kenai. UGO mentors typically participated 
in more than 25 days of professional development during their first two years of mentoring. 
Ongoing support was provided via Friday Forums (online presentations developed by mentors 
on topics of interest to other mentors) and coaching (supporting UGO implementation by 
mentors). Mentors in their third year of mentoring and beyond continued to participate in all 
training events except those specifically required for first- and second-year mentors. 


Anchorage’s mentoring program included up to three trainings: a one-credit graduate-level 
course was required for mentors; a second course was required for dyads post-match; a third 
course was required for dyads in their second year. Each new dyad was required to attend the 
training in Year 1 (and Year 2, if applicable). Anchorage did not provide other ongoing training. 
Mentor training in Kenai varied. When Kenai started the program, mentors attended a four-day 
training followed by a one-day retreat a year later. Mentors participated in monthly meetings 
that included reading and discussing professional literature. They also received calibration 


26 Education Northwest 


training to use the Teachscape program from the Danielson Framework for Teaching (The 
Danielson Group & Teachscape, 2013). 


Interactions with ECTs 


The frequency and duration of mentor-ECT contact was greater in the UGO program than the 
Anchorage and Kenai programs. However, all programs used a standards-based approach and 
incorporated documenting and planning for professional growth. UAF expected UGO mentors 
to maintain weekly contact with ECTs and to meet face-to-face at least monthly (totaling 3.5 
hours). Anchorage mentors were also supposed to document conversations, use a variety of 
tools to guide their mentoring activities, and engage ECTs in reflective processes at least three 
times a year. 


In the first two years of the study, Anchorage encouraged mentors to communicate with their 
ECTs weekly. This was reduced to twice monthly in Year 3. Administrators encouraged twice- 
monthly face-to-face meetings with no minimum time requirements. As Anchorage modeled its 
program on ASMP, they encouraged mentors to document their conversations using a log. 
Mentors could use district-provided tools, open source materials adapted to meet the needs of 
the district, or Danielson-aligned materials. Mentors supported first-year ECTs’ specific needs 
established in a goal-setting process and second-year ECTs’ needs established in a Professional 
Goal-Setting Plan tied to the professional teaching standards. 


In Kenai, administrators expected mentors to have monthly communication, including face-to- 
face meetings, of no minimum length. The Kenai program incorporated coaching strategies 
espoused by Knight (2009), Sprick and colleagues (2009), and Danielson (2010). In the first two 
years of the study, Kenai mentoring focused on practical application and best practice. Mentors 
worked with first-year ECTs to develop a professional development plan and conducted 
monthly check-ins and two observation cycles. Optional activities included ECT observation of 
their mentor and a grade-alike colleague. Second-year ECTs focused on implementing their 
professional development plan. In Year 3, Kenai altered the mentoring model to focus on 
bridging the gap between “knowing” and “doing” (implementation of Danielson’s four 
domains). Mentors, with input from the ECT, developed a coaching plan and activities to 
support a professional goal. 


Other Support 


Analysis of ECT surveys revealed BAU ECTs were statistically significantly more likely to 
report receiving, engaging in, and/or benefitting from some school/district activities that 
supported their early development. These included: 


e Professional development and access to district/content coaches 
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e Formal collaboration opportunities such as common planning time, grade-level 
collaboration, release time to observe other teachers, and support from their site 
administrator 


e Informal collaboration and/or time with other ECTs in their school 


e Access to resources (i.e., materials or volunteers) and university programs 


They were also more likely to experience challenges a mentor might mitigate. These included 
low student motivation, classroom management issues, poor student attendance, and stress. 
Complete survey results may be found in Appendix A. 


Survey Findings 


Survey data on what support UGO ECTs received from mentors revealed differences between 
UGO and BAU ECTs and their mentors. Some findings are understandable, given the 
differences in the formal district-sponsored mentoring programs and the nature of school-based 
or informal mentoring. Others indicate important differences in terms of interaction, topics 
addressed, and alignment with professional and cultural standards. Appendix A contains 
results from ECT surveys administered in spring 2013, 2014, 2015, and 2016 (two years of 
surveys for three cohorts of ECTs). 


UGO ECTs Were More Likely to Have a Mentor but Less Likely to Share Similar Teaching 
Experiences 

Almost all UGO ECTs reported having a mentor, while fewer BAU ECTs did so (99% and 70%, 
respectively). BAU ECTs and their mentors were more likely to have experience teaching in the 
same grade and subject, at the same school level and school, and to have experiences working 
with similar student populations. They were also more likely to have similar teaching 
assignments and to be in close proximity to each other. On the other hand, UGO ECTs were 
more likely to report that they or their mentor were released from instructional responsibilities 
to observe others’ instruction and shared common planning times (we surmise that “common 
planning times” were interpreted as common times during which mentors and ECTs were 
available, since UGO mentors were not teaching). 


UGO and BAU ECTs Thought about Their Mentor’s Roles Differently 
UGO ECTs were more likely than BAU ECTs to think of their mentor as: 


e Anexpert guide (73% UGO and 55% BAU) 

e Role model (72% UGO and 66% BAU) 

e Advocate (66% UGO and 50% BAU) 

e Therapist/counselor (35% UGO and 20% BAU) 


Although UGO ECTs understood the non-evaluative role of their mentor, UGO ECTs were 


more likely than BAU ECTs to consider their mentor as an evaluator and critic (32% and 13% 
UGO compared to 13% and 9% BAU, respectively). Finally, larger proportions of BAU than 
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UGO ECTs considered their mentor to be a colleague (82% versus 69%), which is reasonable 
considering they reported that their mentors were typically teachers in their school. 


UGO and BAU ECTs’ Interacted with Their Mentors Differently 

Larger proportions of UGO ECTs reported communications with their mentor were formal 
(43% and 28%), while larger proportions of BAU ECTs reported communication with their 
mentor was mostly informal (25% versus 3%). This makes sense because UGO mentors 
regularly scheduled visits and interactions with UGO ECTs, whereas BAU ECTs and mentors 
generally had a less formal structure to their interactions. UGO ECTs were more likely to want 
to meet with their mentors during class time, planning time, or lunch (54%, 79%, and 53% 
compared to 14%, 26%, and 17%, respectively). In contrast, BAU ECTs preferred meeting before 
school, after school, or on non-school days (20%, 76%, and 18% compared to 12%, 55%, and 7%, 
respectively). Again, this parallels the UGO model in which mentors are fully released and 
available to meet with ECTs at their convenience. 


UGO ECTs Met with Their Mentors Less Frequently but for Longer Periods 

UGO ECTs most frequently met with their mentors face-to-face every two weeks or monthly 
(89%) for at least one hour (80%); while BAU ECTs met with their mentor more frequently —66 
percent met every two weeks, weekly, or daily —but for no more than one hour (85%) (Tables 9 
and 10). In addition to longer visits, the vast majority of UGO ECTs (96%) indicated working 
with their mentor since the beginning of the school year (August or September); a significantly 
smaller proportion of BAU ECTs did so (81%). 


Table 9. Frequency of face-to-face visits with mentor 
Percentage (n) 


Daily Weekly pri 


BAU 15.9% (57) 15.6% (56) 34.3% (123) 25.1% (90) 8.9% (32) 0.3% (1) 
UGO 0.0% (0) 9.1% (48) 45.3% (240) 44.2% (234) 1.5% (8) 0.0% (0)" 


Monthly Occasionally Never 


Table 10. Duration of face-to-face visits with mentor 
Percentage (n) 


15 minutes or About About About About More than 
less 30 minutes 1 hour 2 hours 3 hours 3 hours 
BAU 24.0% (86) 21.8% (78) 39.4% (141) 12.6% (45) 0.8% (3) 1.4% (5) 
UGO 0.6% (3) 19.1% (101) 34.0% (180) 24.3% (129) 14.2% (75) 7.9% (42)* 


To maintain relationships between face-to-face meetings, UGO ECTs had more frequent 
distance communication (e.g., telephone, email, text) with their mentors compared to BAU 
ECTs: Seventy-five percent of UGO ECTs communicated via these means daily or weekly 
versus 49 percent of BAU ECTs who did so. These findings on duration of mentoring are 
important, as mentoring research suggests a relationship between greater amounts of time 
spent mentoring and positive outcomes. 
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UGO ECTs Trusted Their Mentors More Than BAU ECTs Trusted Theirs 

Although statistically significant, differences between UGO and BAU teachers’ trust ratings are 
small. However, UGO ECTs exhibited a higher level of overall trust with their mentor— 
specifically concerning honesty, benevolence, and reliability —than did BAU ECTs. Our annual 
survey included a trust tool (Hoy & Tschannen-Moran, 1999; Mitchell et al., 2009) that used a 
six-point scale We used ratings of “1” representing “strongly disagree” and “6” representing 
“strongly agree.” Table 11 shows mean trust scores for the tools’ five subscales, each item, and 
the instrument overall. Trust and professional respect is another important factor associated 
with more productive mentor-mentee interactions (Hobson, 2008). 


Table 11. ECTs’ reported trust in their mentor 


Mean (SD)! 
Subscales and items BAU UGO 
Honesty 5.7 (0.59) | 5.8 (0.47)* 
| trust my mentor. 5.7 (0.72) | 5.8 (0.51)* 
| have faith in the integrity of my mentor. 5.7 (0.61) | 5.8 (0.67) 
My mentor keeps his or her word. 5.7 (0.59) | 5.8 (0.50)* 
When my mentor tells me something | can believe it. 5.6 (0.78) | 5.7 (0.70)* 
Benevolence 5.6 (0.64) | 5.8 (0.53)* 
My mentor typically looks out for me. 5.6 (0.85) | 5.8 (0.53)* 
My mentor typically acts with my best interest in mind. 5.7 (0.68) | 5.8 (0.51)* 
My mentor shows concern for me. 5.6 (0.77) | 5.8 (0.57)* 
My mentor is unresponsive to my concerns. 2 5.6 (0.96) | 5.7 (0.97) 
Competence 5.7 (0.64) | 5.8 (0.57)* 
| am suspicious of most of my mentor's actions. 2 5.8 (0.81) | 5.8 (0.84) 
My mentor is competent in doing his or her job. 5.7 (0.74) | 5.8 (0.55)* 
Reliability 5.6 (0.77) | 5.8 (0.56)* 
Even in difficult situations | can depend on my mentor. 5.5 (0.91) | 5.7 (0.67)* 
My mentor is reliable. 5.6 (0.73) | 5.8 (0.55)* 
Openness 5.2 (0.90) | 5.2 (0.76) 
My mentor is open. 5.7 (0.70) | 5.8 (0.51)* 
My mentor openly shares personal information with me. 4.8(1.44) | 4.6 (1.36) 
Total score 5.6 (0.59) | 5.7 (0.45)* 


1 We reversed the scale used by Hoy and Tschannen-Moran(1999) (“1” represents “strongly agree” and “6” 


represents “strongly disagree” to align with other scales in which higher numbers indicate higher levels of agreement. 
2For reporting purposes, negatively worded items were reverse coded so thata higher rating consistently indicates a 


more positive attitude. 
* 9 £0.05 on independentt-test. 


UGO ECTs Received Support from Their Mentors, in a Variety of Areas, Significantly 


More Often Than BAU ECTs Did from Theirs 


Table 12 displays specific activities and discussion topics in which mentors and ECTs could 


engage. In all of the areas, UGO ECTs reported activities and discussions occurred significantly 


more frequently than did BAU ECTs (p< 0.05). 
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Table 12. Frequency of statistically significant mentor/mentee activities and discussion topics 


Percentage (n) 


Every two 
Never/ weeks/ 
Group _ occasionally Monthly weekly 


Activities’ 
. BAU 89.1% (317) 7.9% (28) 3.1% (11) 
Obscrye vOURINS EURO UGO _ 11.4% (60) __52.3% (276) 36.4% (192)* 
Bae nese Gas BAU 84.2% (298) 10.5% (37) 5.4% (17) 
late UGO 24.4% (129) 44.4% (235) 31.2% (165)* 


( ( 
——— ( ( 
Model lessons or strategies with BAU 92.7% (330) 4.2% (14) 3.1% (11) 
( ( 
( ( 


your students and/or co-teach in ‘ 
your classroom UGO 79.7% (408) 12.1% (62) 8.2% (42) 
70 


Provide you with resources and BAU 44.3% (158) 19.6% (70) 36.1% (129) 

materials UGO 24.3% (129) 29.4% (156) 46.2% (145)* 

BAU 43.1% (154) 19.3% (69) 37.5% (134) 

Poctimen yeur wor together UGO 7.0% (37) 34.0% (180) 59.1% (313)* 
Engage with you in goal setting BAU 56.9% (203) 20.7% (74) 22.4% (80) 

rocess UGO 15.5% (82 42.3% (224 42.2% (223)* 


Brainstorm with i 
eri ee apeadh i Acicmete Ata BAU 30.1% (107) 18.5% (66) 51.4% (183) 


approach a challenge with a 
ane UGO 11.2% (59) 29.9% (158) 59.0% (312)" 


Discussions 
Observation of your instruction BAU 77.5% (276) 11.8% (42) 10.7% (38) 
and/or data that were gathered UGO 11.1% (50) 46.3% (208) 42.5% (191)* 
: BAU 46.5% (166) 21.6% (77) 39.1% (114) 
pe les obeauily UGO _ 18.9% (100) _ 39.1% (207) _ 42.1% (223) 
Cultural awareness, values, and BAU 67.2% (240) 14.3% (51) 18.5% (66) 
sensitivit UGO 44.2% (234 31.0% (164) 24.8% (148)* 
j ; ; ; BAU 51.5% (184) 19.3% (69) 29.1% (104) 
Working with special populations Gq 38.5% (204) 28.9% (153) 32.6% (173)" 
; BAU 54.1% (193) 20.5% (73) 25.5% (91) 
Peso bionnig UGO _ 47.4% (249) 28.2% (148) 24.4% (128)* 
ieee BAU 63.9% (228) 18.8% (67) 17.4% (62) 
Paentcommunicaeh UGO __54.5% (289) _ 28.3% (150) _17.2% (91)* 
Site administrator/principal BAU 66.4% (237) 15.4% (55) 18.2% (65) 
communication UGO 57.1% (302 25.7% (136 17.2% (91)* 
1 Using chi-squared, UGO ECTs were found to reportall activities and discussions occurred significantly more 
frequently than did BAU ECTs (ps 0.05). 


During mentoring sessions, compared to BAU ECTs, larger proportions of UGO ECTs and their 
mentors addressed issues related to classroom management (94% versus 88%), pedagogy (76% 
versus 55%), and district logistics and procedures (64% versus 56%). These are important 
findings as they reinforce other research studies on mentoring that identify the beneficial role of 
formative feedback cycles and educative mentoring practices (Feiman-Nemser , 2001). 


UGO ECTs Were More Likely to Indicate Their Mentoring Was Standards-Based and 
Culturally Relevant 
Compared to BAU ECTs, UGOECTs were more likely to strongly agree that their work with 


their mentor was: 
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e Standards-based: guided by professional teaching standards (76% UGO versus 57% BAU) 
and included content, performance, and cultural standards (62% and 41%) 


e Culturally relevant: supported their understanding of the culture(s) in their community 
(56% UGO and 48% BAU) 


Summary of Differences between the Experiences of UGO and BAU ECTs 


The UGO and BAU conditions differed in numerous ways. While ECTs in both conditions had 
access to mentors, Anchorage and Kenai mentors had lower experience requirements and 
participated in less initial and ongoing training. Anchorage mentors had low caseloads (usually 
one ECT per mentor); however, mentors retained all of their teaching responsibilities. Kenai’s 
fully released mentors typically had caseloads of about 14 ECTs. In both districts, at least 
monthly contact was encouraged, but program administrators required no minimum contacts 
(face-to-face or otherwise). Possibly because BAU ECTs did not have UGO mentors, district and 
school administrators might have provided them access to other supports to encourage their 
early development, or BAU ECTs might have felt they benefitted more from receiving them. In 
districts without formal mentoring programs, these supports often included access to some type 
of content coach. 


The experience of ECTs with regards to professional support during their first two years of 
teaching was different for the UGO and BAU ECTs. BAU ECTs had more in common with their 
mentors than UGO ECTs. This was influenced by the Anchorage mentoring program, which 
matched mentors and ECTs more closely than the ASMP program did. Additionally, school and 
informal mentors were more likely to be colleagues, working in similar situations that put them 
in regular contact with each other. Differences in the nature of these relationships likely 
impacted the ways ECTs viewed their mentors and the formality of their communication. 
ASMP’s fully released mentors could work with ECTs during the school day and for longer 
periods of time. The fact that they were balancing caseloads likely required them to spread 
contact out to monthly or bimonthly visits. Mentors of most BAU ECTs in the Anchorage 
program or school-based mentoring taught full time and could only work with their ECTs 
outside of the school day.? School and informal mentors likely met with ECTs in a more 
impromptu manner as they interacted with each other during the school day or work week. 
Therefore, their meetings were more frequent, but shorter. 


BAU and UGO ECTs engaged in similar activities with their mentors, but UGO ECTs 
consistently did so with more frequency. This could stem from a variety of reasons, including 
differences in the time availability of UGO mentors, ASMP mentor-ECT contact requirements, 
UGO mentors’ additional classroom experience, and their training and professional 
development. The latter encouraged and prepared them to engage in a variety of practices ona 
regular basis, including standards-based goal setting; instructional observations with data 


2 Tt is important to note that while a component of the Kenai mentoring program used fully-released 
mentors, this comprised a small number of ECTs in total. 
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collection and debriefing; and conversations focused on equity, diversity, and problem solving. 
While all three formal mentoring programs did use a standards-based approach, the ASMP 
training particularly emphasized linking all conversations toa variety of standards, building 
relationships, and responding to needs expressed by their ECT. With their training, UGO 
mentors reportedly addressed standards more consistently; were better able to engage with 
their ECTs; built more trust; and, ultimately, impacted ECT practice in areas meaningful to their 
ECTs (p<0.05). 


ECTs in the BAU and UGO conditions underwent substantially different experiences in their 

first two years of teaching. BAU ECTs were more likely to report receiving, engaging in, and/or 
benefitting from some school and district activities that supported their early development and 
were more likely to experience some challenges a mentor might mitigate. UGO-mentored ECTs 
reported significant impact on a broad range of mentor-mentee activities and discussion topics. 
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Chapter 3. Implementation Study 


Key aspects of the National Evaluation of i3 (NEi3) guided how we measured implementation. 
This included ensuring that the study was well-designed, well-implemented, and independently 
conducted. NEi3 guidance in accordance with high standards of educational research also 
indicates that implementation studies should provide information about the key elements and 
the approach of the project, which facilitates replication or testing in other settings. This chapter 
presents our findings from the implementation study conducted during the first three years of 
UGO, from summer 2012 through spring 2015. 


Research Questions 


To assess how well UAF implemented UGO as planned, we developed four questions that were 
aligned with the actions of the project and focused on the project’s key components, as included 
in the logic model: 

1. To what extent were key components—mentor recruitment/assignment, mentor 
participation in professional development, mentor interactions with their ECTs, and 
mentor use of formative assessment tools—implemented as planned? 

What conditions facilitated or challenged implementation? 
How much variation in implementation was there across mentors? 


In what ways were key components implemented differently from the model as planned? 


Methods 


Education Northwest researchers worked with UAF researchers and administrators to develop 
a logic model. Together, using the logic model as a basis for measuring implementation fidelity, 
we created a fidelity of implementation (FOI) matrix. The FOI matrix includes stated 
expectations from UAF (indicators) for implementation of UGO’s four key components. We 
describe these key components and indicators in appendix B. 


In addition to the key components and indicators, the matrix also identifies three levels of 
implementation—low, adequate, and ideal based on implementation of the ASMP model. We 
established numeric thresholds for low (“0”), adequate (“1”), and ideal (“2”) implementation: “0” 
represents the absence or unacceptable implementation of indicators. A “1” represents adequate 
implementation of the indicators and is the minimal expectation. A rating of “2” represents 
meeting the criterion for ideal implementation of the indicators. 


We scored all UGO mentors in all five districts on each key component's indicators. We then 


used the results of the mentor-level scores to calculate the percentage of mentors with fidelity 
scores of “1” or “2” foreach indicator. To determine key component fidelity, we averaged the 


Mentoring ECTs in Urban Alaska 35 


percentage of mentors with fidelity scores of “1” or greater across indicators. We considered the 
component implemented with fidelity if the average of all indicators was at least 70 percent of 
mentors scoring “1” or greater and at least 51 percent of mentors scored “1” or greater for each 


indicator. If these two criteria were not met, we considered the component not implemented 


with fidelity. 


We used qualitative and quantitative 
strategies to gather data on the four key 
component areas of implementation 
(shown in Figure 1 and in the sidebar). 
We conducted interviews with mentors, 
district coordinators, and UAF 
administrators (including professional 
development trainers and coaches) to help 
interpret findings from analyses of the 
FOI matrix and to answer the four 
research questions. 


Appendix C contains tables describing 
project-level findings for each of the 16 
indicators included in the four key 
components of the UGO FOI matrix. 
Appendix C also contains an overview of 
the FOI tables. 


Findings 


Below we report our fidelity of 
implementation findings, by key 
component with information on 
indicators. (See Figure 1 or the sidebar for 
indicators associated with key 
components.) We also include facilitating 
conditions and challenges stakeholders 
experienced during implementation. 
When applicable, we address differences 
in implementation across mentors. 
Finally, in each section we document the 
overall differences we identified between 
AK DEED’s implementation of the ASMP 
model and UAF’s implementation of the 
UGO model of the mentoring project. 
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Key Components and Indicators from Logic 
Model Used in FOI 


ASMP UGO guides district in the recruitment 
and assignment of highly qualified mentors 

e Experienced teachers (at least 8 years in Alaska) 
e Full-release mentoring: dedicated time for 
mentoring 

e Caseload of no more than 15 ECTs per mentor 


Mentors participate in in-depth professional 

development 

e Fully attend Orientation (~2 days/year) 

e Fully attend Wrap Up (~3 days/year) 

e Fully attend mentor training, using the ASMP 
approach/materials based on the New Teacher 
Center (NTC) model (4 academies/year for 2 
years) 

e Fully attend ASMP training (3-4 sessions/year) 

e Participate in Friday Forums (at least 10/year) 

e Communicate monthly with coach 

e Fully participate in shadowing sessions (2 
sessions as first-year mentor and at least 1 in 
subsequent years as a mentor) 

e Participate in coaching, using mentor formative 
assessment tools 


Mentors interact with ECTs 

e Communicate at least weekly with ECTs 

e Provide face-to-face interaction (at least 
3.5 hours/month) 


Mentors use formative assessment tools with 
ECTs* 
e Document conversations through Collaborative 
Assessment Log 
e Use formative assessment tools to support 
ECTs and gather classroom data 
e Support reflective practice through Individual 
Learning Plan, Mid-Year Review, and 
Professional Growth Reflection 


*Note: Connect all work to Standards for AK 
Teachers, AK Cultural Standards, and/or Continuum 
of Teacher Development. 


Mentor Recruitment and Assignment 

In all three years, UAF implemented this key component with fidelity. At the indicator level, 
ideal fidelity was more likely to be reached in hiring mentors with appropriate teaching 
experience and providing dedicated mentoring time; maintaining ideal caseloads was slightly 
more problematic. Across the three years, we found one instance in which a district hired a 
mentor with less than eight years of teaching experience in Alaska and two instances in which a 
district assigned mentors to non-teaching duties in addition to their mentoring. UAF, however, 
found maintaining ideal caseloads of 12-15 ECTs for full-time mentors slightly more 
problematic. While most district coordinators assigned mentors ideal caseloads per their full- 
time equivalent (FTE), when this was not the case, mentors were more likely to have smaller 
than larger caseloads. 


Prior experiences, existing and new relationships, and a willingness to collaborate facilitated 
implementation. District coordinators said their knowledge of district policies, practices, and 
staff members helped them effectively engage in their work to hire mentors, identify ECTs, and 
assign caseloads. Districts hired district coordinators, who were long-term employees that 
understood the complexities of their district and its policies. Over the course of the three years 
of measured implementation, district coordinators underscored the importance of having deep 
knowledge of their districts and internal processes and having existing professional 
relationships within the district, as well as an openness to building new relationships. Working 
with various district and school contacts, human resources, and teachers’ unions provided them 
access to potential hires (both mentors and ECTs), mentor interviewing and contracting 
support, and venues that allowed them to recruit ECTs and obtain their consent to participate in 
the study. 


Likewise, mentors reported that the position was a “good fit” for them because they had 
previously been ASMP mentors, were long-time district employees who knew district policies 
and professional contacts, and/or were experienced mentors or professional development 
providers under other models. 


Last year I was a district coach. I went through the application process previously for the 
district. I had all of last year to interpret what I was doing as a coach. I was working in a 
school with nine teachers. I kind of lived with them in the same building (I had previously 
taught in for 20 years). I was a resource for field trips, community connections, how to do 
things within the district. (Mentor) 


UAF staff reported sharing the AK DEED mentor application, interview protocol, and scoring 
rubric with districts and district coordinators and helping them think about who in their 
districts might be a good fit for the UGO model. They also mentioned the importance of being a 
good listener. This helped the UAF staff, district coordinators, and researchers to develop close 
partnerships and allowed the team to truly problem solve, as all parties felt comfortable being 
open and honest in voicing challenges and concerns. 
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The rubric, application, and questions that UAF shared were really helpful. We modeled 
our hiring after that, but incorporated many district pieces into the process. I spend some 
time talking with those who deal with contracts and purchasing. We incorporated the 
standard procedure of hiring during the interview process. Even though it is a contract 
position, it isa good model and we thought it would feel familiar to the many district 
retirees who were applying for the positions. So we began with UAF materials and 
morphed it to fit district policy and procedures. (District coordinator) 


District coordinators also mentioned the importance of communication among district 
administrators (including principals). Keeping everyone informed about the model and 
implementation in the district was key. Similarly, having existing relationships with district and 
school administrators and educators aided in communication, for both mentors and district 
coordinators. 


Not knowing the number of ECTs districts would hire each school year made hiring mentors 
and assigning caseloads challenging; this was exacerbated by restrictions on the number of 
hours districts could hire mentors to work. While human resource departments were partners 
in identifying and recruiting ECTs, these systems were not perfect. Some district coordinators 
had trouble predicting the full number of vacant positions and to what extent they might be 
filled by ECTs versus experienced teachers. The former challenge varied depending on district 
size. 


Our district is small enough that I have a staffing changes sheet in front of me at all 
administrator meetings. We go over [staffing needs] in spring with transfers to 
buildings, retires, and new hires. We basically keep a running record of that. (District 
coordinator) 


Some district coordinators also had difficulty identifying ECTs once hired. Not all districts 
tracked this in their hiring process and venues for bringing together new hires (e.g., orientation) 
varied or were nonexistent. 


Last year we had a new hire orientation meeting that allowed us to identify ECTs. That 
meeting was cancelled this fall, so it was a huge challenge identifying ECTs. I had a lot of 
meetings with human resources to figure out how they identify them (the ECTs in the 
new hires). We tried a couple of different ways to look at the data they collected, but every 
time our lists were never clean. Now I have access to the application software and can 
match against other documents. (District coordinator) 


In hiring mentors, district coordinators had to meet the various requirements of UAF, their 
district, teachers’ union, and state. AK DEED hires only full-time ASMP mentors to meet the 
needs of ECTs in rural districts. However, in UGO, various situations in the urban districts (e.g., 
limits on the number of hours retired or independent contractors could work) required UAF to 
adjust the model to include the use of part-time mentors with smaller caseloads. Ultimately, 
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UGO mentors held mentoring positions that varied in FTEs, including full-time regular 
employees, part-time regular employees, and full-time and part-time annually contracted staff. 


Estimating the number of ECTs districts would ultimately hire by fall, in the summer, was 
necessary to allow districts to hire adequate numbers of mentors in time for them to participate 
in required summer training sessions. If these estimates were off, district would hire—and UAF 
would train—too many or too few mentors for temporary or permanent part-time or full time 
positions and assign caseloads that could be too large or too small. Some years, districts needed 
to hire additional mentors, who UAF then trained after the start of the school year. Some 
mentors went from part-time to full-time positions, and caseloads typically changed to 
accommodate late district hires. Defining appropriate caseloads for varying full-time 
equivalency positions was an additional challenge. 


At one point, the district coordinator was unsure if she could hire all of us. Some mentors 
were already hired full-time so they asked me if I would consider part-time. (Mentor) 


Balancing caseloads was sometimes tricky because of ECTs’ differing years of experience and 
teaching settings. Usually, ASMP mentors have full-time caseloads of at least 15 first- and 
second-year ECTs assigned to a variety of positions (e.g., elementary, secondary, general 
education, English language arts, mathematics, and science) and locations (multiple districts). 
Some district coordinators tried to assign UGO mentors caseloads that matched their 
background and experience, but they were not always successful in doing so. 


[I] tried a little to give mentors a geographic caseload, but that was difficult because I did 
not know where the next batch of ECTs would be. (District coordinator) 


Mentors underscored the challenge. Generally, they felt their caseload was “just right.” They 
reported their full-time caseload was most reasonable when it included 12 to 15 ECTs. It was 
easiest when the majority of their caseload included second-year ECTs. Finally, it was hardest 
when they had several particularly needy first-year ECTs, ECTs at many different locations, or 
ECTs who taught in an area or level with which they had less experience. 


Twelve is good for this year, because I have at least four that are second year. Last year, 
with just first-year teachers, it was a drain emotionally. First-year teachers just need 
more attention. Not every first-year teacher needs a lot of attention, but generally they 
need more of my time, more of my encouragement. This year it’s a little bit more 
balanced. Maybe that’s because I’m getting it, too. (Mentor) 


We had other discussions about matching by subject, levels of expertise, geographic 
considerations. We tried to divide it up that way, even though it does not always work. 
Even though there are three ECTs in one school, it does not mean you can meet them all 
the same day. (Mentor) 
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Mentor recruitment and assignment differences between ASMP (rural) and UGO. The ASMP 
model was developed and refined in response to needs of ECTs in village/rural schools in 
Alaska. Of particular interest to UAF are collateral findings from this study related to 
differences between ASMP, as traditionally implemented in predominately rural districts, and 
UGO, implemented in urban districts. While UGO adopted the key components of the ASMP 
model (as described in the logic model), adaptations were made to accommodate the urban 
setting. With this context in mind, we identified some areas in which UAF implemented the key 
component of mentor recruitment and assignment differently from the model as planned (e.g,, 
AK DEED’s implementation of ASMP): 

1. Individual districts hired and contracted with UGO mentors, whereas AK DEED hired 

and contracted with ASMP mentors. 


2. UGO mentors could work varying levels of part-time, with appropriately reduced 
caseloads. ASMP mentors were only full-time. 

3. District coordinators more frequently matched UGO mentors to ECTs based on grade 
level, school level, or content-area experience. This was especially true in the area of 
special education. This is not an AK DEED practice. 

4. UGOserved a much broader array of ECTs, including physical education, music, and 
shop teachers, as well as teachers hired to provide instructional interventions to small 
groups of students. In rural/bush settings, a small academic team often teaches these 
instructional areas, and traditional academic teachers make up the vast majority of ECTs 
with ASMP mentors. 

5. UGO mentors benefitted from having a district coordinator who had ready access to 
school and district administrators to enhance the visibility of, and support for, mentoring 
at the district level. 


Mentor Participation in Professional Development 

In all three years, UAF implemented this key component with fidelity. At the indicator level, 
ideal fidelity was more likely to be reached in regard to mentors participating in professional 
development events than in coaching activities. Full participation (attending all scheduled 
sessions) in professional development events such as Orientation, NTC (Academy) and ASMP 
training, Friday Forums, and Wrap Up was attainable for most mentors most of the time. When 
mentors missed these events, they were more likely to not attend rather than partially attend 
these events. They were most likely to miss Orientation. 


Documentation on UGO coaching activities suggests UAF’s implementation of coaching was 
less than ideal. Coaching includes shadowing mentors, communicating with them monthly, and 
engaging them in reflective practice. We have more evidence that shadowing occurred, because 
we supplemented coaches’ documentation with mentor reports during the annual interviews. 
However, based solely on coaching documentation, implementation of the other coaching 
activities varied. In Year 2, coaches engaged in monthly communications more frequently than 
in other years. In Year 3 they participated in reflective practice activities more frequently than in 
previous years. 
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Mentors appreciated the project’s high-quality training. District coordinators, mentors, and 
UAF staff members all commented on the quality of the training. Mentors reported the 
professional development was high quality and provided them the necessary time to learn 
about and practice using mentor language and tools. Several appreciated the variety of formats 
used. The Academy sessions were especially valuable because they incorporated small- and 
whole-group time, modeling, and role-playing, and afforded time to learn from other mentors. 
This time to socialize, network, and engage in small-group work was especially valued in terms 
of support and camaraderie. 


It is fabulous. That is the best word I can come up with. It is not just the content. It is 
being able to physically get together with the other mentors. Besides working with them 
in the training, it is even outside of the training when you are talking about your ECTs 
and their issues and strategies and resources. It is just great. Knowing there are other 
people out there doing the job; you do not feel like you are alone. It’s invaluable training. 
The content is really, really good. All of the tools we are receiving have been invaluable. 
Just the support. It is great to havea coach who is there for you and a partner who is 
there for you. (Mentor) 


Because mentors perceived the training as so worthwhile, none wanted to miss out. 


It’s just such good training. I never have to say they have to come. (District 
coordinator) 


Mentors appreciated the support they received from their peers and peer coaching partners. 
Many mentors appreciated the coaching partner aspect of the project and saw it as more 
conducive to their learning and needs than the regular coaching component. With coaching 
partners, UAF assigned mentors a mentor colleague who often lived in a geographically similar 
area and was a first line of support (first coaching partner, then coach, then UAF 
administrators). Mentor partners were regularly used in Friday Forums in Year 3, but were a 
less-emphasized, regular part of the mentoring model from the beginning. 


Fabulous, they can say things to each other that no one else knows about. It is very 
helpful. Coaching partners provide someone to talk with about things that are tough. 
They have each other’s backs. (Mentor) 


While most mentors felt supported by their coaches and reported coaches had fulfilled the 
expectations of the model, it was the most unevenly reported aspect of the professional 
development package. This was reportedly due to mismatching coaches to mentors and varying 
district policies regarding visitors in schools. 


The first time my coach shadowed me, they took data. That was what propelled me to be 
more confident about taking data. That was very helpful. I have one teacher in particular 
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that is pretty openly resistant to mentoring and has been pretty upfront about it, which is 
fine. My coach has helped me think about ways to work in that relationship, to be helpful 
but not aggressive ... I was just searching for the email this morning. My coach sent me 
an excerpt from an email of an interaction with a [similar ECT]. I really appreciated that. 
It was personal. I think when you share your personal experience with someone who is 
struggling, it’s helpful. (Mentor) 


Scheduling issues affected training attendance and mentors’ work with each other and their 
ECTs. Discrepancies between mentors’ hire dates, previously planned trips, and other 
commitments and scheduled professional development dates caused some mentors to miss 
required training, most often Orientation, the first Academy, and shadowing. The multiday 
training events took mentors away from their regular work schedule and made it difficult for 
them to engage, as expected, with their ECTs. Part-time mentors were especially challenged 
with the amount of training required. 


[Professional development should be scheduled] as early as possible at the first of the year 
to get the dates for the next year set. Sometimes travel plans [of potential mentors] are 
made way in advance. We wanted the training dates set in the application for mentors 
before we posted it, but that did not happen fast enough. (District coordinator) 


UAF’s effort to reduce costs in Year 3, by moving the in-person training events from Fairbanks 
to Anchorage, resulted in shortened sessions. The condensed training time also reduced contact 
time between mentors because there were fewer days of training that were more tightly 
scheduled. With fewer training days there was less collaboration time, either formal or informal. 
To some extent, some districts might have mitigated the challenge of mentor-mentor 
communication by arranging meetings outside of the formal UGO trainings and providing 
office space for mentors to meet informally. 


Some training content was repetitious and not all training formats met the needs of attendees. 
Some attendees reported that training content was at times repetitive or less applicable to their 
work. Mentors specifically reported concerns about repetition in the second set of the four NTC 
academies. Mentors also perceived Friday Forums in Years 1 and 2 as less applicable to their 
work. This prompted changes in the structure and content of these sessions to address concerns. 
In rating and discussing the professional training, mentors made comments such as: 


Excellent. Academy 1-4, 5 and 6 —relevant to what we do, pace is just right, no lag time, 
presenters are well-prepared, respectful of their professionalism ... I wonder if there is 
enough information to fill eight academies. (Mentor) 


Average; it is a lot of repetition. The first two years we had the full academies; once we 
are done with the academies I do not feel a lot of it is necessary. What is necessary is the 
face-to-face contact with the other mentors, which we don’t get this year; being able to tap 
other experts in the area is very valuable, now I feel it’s busy work. So same or more time 
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with other mentors but less busy work with them. In Fairbanks academies, the best PD 
was just talking to each other outside the formal activities. This is totally missing from 
the project now. (Mentor) 


UAF staff members and trainers spoke to the importance and challenge of delivering content 
geared to the experience of the mentors (first year, second year, third year, or more) and 
keeping the training relevant and engaging for all participants. 


Finally, we identified one area in which UAF and UGOimplemented the key component of 
mentor participation in professional development differently from the model as planned (i.e., 
AK DEED’s implementation of ASMP). As UGO districts had multiple mentors in close 
proximity to each other, some UGO mentors had additional opportunities to meet and share 
with each other, formally and informally. This was not designed into the model, but was 
appreciated by UGO mentors as an additional professional support. 


Mentor Interactions with Their ECTs 

In all three years, UAF implemented this key component with fidelity. At the indicator level, 
ideal fidelity was consistently reached in regard to maintaining weekly communication. 
Mentors less frequently accumulated 3.5 hours of face-to-face interactions with their ECTs. 
With the exception of some ECT who took leave or discontinued mentoring during the school 
year, all mentors maintained weekly communication with their ECTS. Some mentors struggled 
to accumulate 3.5 hours of face-to-face interactions with their ECTs. In all years, the mentors 
who did not accumulate 3.5 hours of face-to-face time were evenly distributed across most 
districts. Some mentors consistently failed to use (or document their use of) tools with their 
ECTs. 


Developing routines was an important aspect of implementation that aided mentors in meeting 
the project’s contact requirements. Routines allowed mentors to be persistent, consistent, and 
efficient. 


In my calendar, on one side I have a vertical chart with all my ECTs and then every week 
I have highlighted and noted if I’ve emailed them, if I have a face [-to-face] visit with 
them, a phone call to them. I circle if they responded. Then I can see at a glance who has 
had a face visit, who has emailed, etc. It is very easy for me to see who needs attention. I 
am not leaving anybody out. (Mentor) 


IT have been very clear from the beginning with my ECTs about the expectation and what 
we do. I have gotten better at emails; learned to give them short email that asks specific 
things about their practice. I take notes during the observation and include follow-ups on 
that in emails. (Mentor) 


Flexibility in implementing the model helped overcome a challenge related to ECT’s tight 
schedules. Mentors took their jobs seriously and did their best to meet all the project 
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expectations. However, their work—and that of their ECTs— pulled them in many directions, 
which made it difficult to find time to communicate. UGO mentors appreciated that their ECTs 
had a lot on their plates and could not always find an hour a week, two hours every two weeks, 
or four hours a month when they could work face-to-face with their mentor. They understood 
that ECTs sometimes could not give up their planning time to meet and that other 
school/district requirements (e.g., professional development, professional learning 
communities, team planning, union contracts) took priority. They also realized that ECTs had 
additional responsibilities that occurred after school—some related to school (e.g., coaching) 
and others related to their personal lives (e.g., children). Mentors said that flexibility and being 
realistic about what they could accomplish in their day-to-day work were factors that helped 
them implement the model. 


[A challenging area] on site visits is pre- and post-observation discussions. In reality, 
ECTs rarely respond that the time I am coming is going to work, let alone what they 
would like me to focus on, and they do not have time after school to do it, either. Few have 
lesson plans to look at in advance. A lot of the post-observations discussion at the 
elementary level happens informally between the observation and the time when we do 
the CAL. Many of my elementary ECTs have centers, so we have a lot of time when they 
are monitoring to touch base on what works and does not work. It happens in bits and 
pieces rather than in one big chunk. It’s good discussion, with their think-alouds and my 
questions and dialoguing in between things, and it’s building a relationship with them. 
As a result, that is my reality versus the structured pre/post discussions. (Mentor) 


Finally, an important aspect of flexibility included finding ways to communicate with and 
respond to their ECTs at times, and in ways, that were most convenient and meaningful to the 
ECT. 


Finding chunks of time to talk with ECTs. Some only want to put in their contracted 
time and it’s hard to get time with them; others are willing to meet outside of school at 
restaurants, coffee shops. It’s hard to find time to meet around class, lunch, and planning 
time. (Mentor) 


[We were told to meet face-to-face with our ECTs] once a month. I see all of them (but 
one) two, three, sometimes four times a month. Face-to-face visits are just as easy as an 
email. My ECTs prefer it. I have three men that are terrible about email. I can more easily 
visit. (Mentor) 


District coordinators appreciated that their mentors could be flexible with their time to best 
meet the needs of the ECTs they supported. Being full-release mentors definitely aided them in 
doing so. 

Training supported effective mentoring. We specifically asked mentors about the extent to 


which the professional development prepared them to work with their ECTs, and mentors 
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overwhelmingly agreed that it did. It provided them with increased ability to find entry points 
in conversations with ECTs and to use sentence/question “stems” and a framework for 
engaging in appropriate instructional, collaborative, or facilitative conversations. Training gave 
them knowledge about how the different tools worked and practice in using them before going 
out in the field. It also highlighted the importance of integrating reflection into their work with 
ECTs ona regular basis, not just when using the designated reflective practice tools. 


I learned to do the tools using them in a variety of ways, practice and role playing, 
reflective conversations, instructive, facilitative, collaborative role. This mentoring 
training taught me the reflection model and that I’m going to weave in and out between 
the three instructive, facilitative, collaborative roles—that has been very helpful to me. 
The academies build on each other and that message has come through. I’m using the 
three different models and integrating them with reflection. (Mentor) 


Finally, we identified some areas in which UAF and UGO implemented the key component of 
mentor interactions with ECTs differently from the model as planned (i.e., AK DEED’s 
implementation of ASMP): 


1. 


UGO mentors had the opportunity to engage in more frequent face-to face interactions 
(which is more like the NTC model than the ASMP model used in rural settings), but 
they often found it difficult to find the time to doso. 

ECT/mentor relationships may focus more on instructional issues in UGO. In contrast, 
they might include more personal and social aspects in the rural settings, where mentors 
often spend overnights or multiple days on site. 

UGO mentors supported a broader array of cultures in urban settings, rather than the 
Alaska Native populations that are the focus of ASMP mentors. 

UGO mentors were often teachers who had retired from the district they mentored in and 
only mentored in that one district. This provided them the opportunity to support their 
ECTs in targeted district initiatives. ASMP mentors work across multiple districts and 
often to do not have the same in-depth knowledge of various district initiatives. 

To learn about district initiatives, ASMP mentors intentionally developed relationships 
with their ECTs’ site administrators. This was a less-emphasized component of the UGO 
model. Many UGO mentors reported that gaining access to site administrators was 
difficult, especially at the secondary level. 

Urban schools have more supports/resources available to ECTs than village/rural schools. 
UGO mentors worked in one district, while ASMP mentors work across multiple 
districts. Meeting communication requirements was sometimes difficult for UGO 
mentors who could only juggle one district calendar, as compared to ASMP mentors who 
can juggle multiple calendars. For example, a vacation day in one district might be an 
instructional day in another district, allowing the mentor to visit that ECT while school is 
in session. 
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Mentors Use of Formative Assessment Tools 

In all three years, UAF implemented the use of formative assessment tools with fidelity. Ideal 
fidelity was consistently reached in using Collaborative Assessment Logs (CALs) to document 
mentors’ weekly communications with their ECTs and engage them in reflective practice. 
Using, or documenting their use of, tools was less likely to be at ideal levels. With the 
exception of some ECTs who took leave or discontinued mentoring during the school year, 
almost all mentors completed CALs and used the project’s reflective practice tools with all of 
their ECTs. Some mentors struggled to use (or document their use of) other formative 
assessment tools. In all years, the mentors who did not use (or document their use of) tools were 
evenly distributed across most districts, but some mentors consistently failed to use (or 
document their use of) tools with their ECTs. 


Relationship-building was a key aspect of engaging with ECTs. Mentors discussed the 
importance of establishing relationships with ECTs as the prerequisite of changing practice. 


[In mentoring] part is personality, part is previous experience, part is counseling. The 
most import component is the relationship. I just focused on building relationships. I am 
pretty sure all my ECTs value our relationship. It is not just having a mentor, it is 
someone you trust is confidential [and who] you can count on. I can have difficult 
conversations with them. (Mentor) 


Mentors’ ability to rely on professional judgement and be flexible in their work facilitated tool 
use. District coordinators, mentors, and UAF staff members appreciated that mentors could use 
NTC, ASMP, and district tools and that having a larger toolbox provided them with the 
necessary data they needed to help ECTs see what was happening in their classrooms. 


I supplement the tools. There are a certain set of tools the district uses, similar but easier 
to use then the ASMP tools. In 15 minutes, I can collect a variety of these data (i.e., 
engagement, positive ratio of interactions, and opportunities to respond). (Mentor) 


Six of my ECTs teach students with disabilities. Some of the tools are not formatted for a 
nonacademic setting. The curriculum is functional tools in a real-life setting. (Mentor) 


Mentors also mentioned ways that they improvised their data collection, often on the fly. 


Some of the teachers I work with are kind of different or have different settings. Instead of 
using the actual tool, I have taken data on a piece of paper, and I have modified the way I 
take the data to the environment and to the teacher. I just sit down with the teacher and 
say, “Today, I took engagement data and this is what I came up with. How does that look 
to you? What are you thinking?” Try to get a conversation going from there. (Mentor) 


Finally, UAF staff members discussed how effective managing helped ensure fidelity of 
implementation. During shadowing, coaches observed the use of tools and the frequency of 
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communication between mentors and ECTs. They also reviewed mentors’ documentation of 
their work to ensure mentors were appropriately contacting their ECTs. They shared this 
information at management meetings, and if needed, UAF administrators were available to 
work with mentors to address any concerns. 


Training supported implementation. As discussed earlier, we specifically asked mentors about 
the extent to which the professional development prepared them to work with their ECTs, and 
mentors overwhelmingly agreed that it did. Training provided time to practice and explore 
how they could use different tools in different situations. 


Data collection is in the training. We customize the data collection to the teacher and you 
need to find out what that teacher needs and the training allows us to find ways to help 
teachers think about their practice. (Mentor) 


UAF staff members echoed the value training and coaching provided for mentors to effectively 
interact with ECTs and use tools. 


Mentors found the project’s formative assessment and reflective practice tools valuable. While 
many mentors commented that documenting their work was a time-consuming part of their job, 
many saw the value in it, especially when it was done collaboratively with their ECTs. 


I work very hard on it. It takes me at least 1 1/2 hours per ECT each week. I want it to 
connect to the standards. I want it to be useful to the ECTs. I want it to be something 
they can show to their administrator. I want the ECT to know they have been heard. The 
different tools let us show the ECTs the amazing things they are doing. I have second- 
year teachers, and so much of what they are doing —the tools let them see that not only 
did they accomplish their goals [but they also] addressed other issues. It helps them see 
their growth. (Mentor) 


The tools definitely help to open up conversations. I feel like I’m definitely moving them 
forward, working with them where they are at and having them discover what do to do 
next. The tools are a great eye-opener and no one feels threatened by them. (Mentor) 


In addition, they noted the formative assessment tools provided immeasurable support in 
engaging with ECTs to move their practice forward. 


Tf you ask: What's working? You get “Everything is fine.” If you ask: Any challenges? 
You get, “No.” If they feel like its evaluation you do not get anywhere with them. But, if I 
link my comments to observable behavior then they open up; you are not judging them. 
Observable behaviors that we can present to them in a non-evaluative way — students 
were talking or you needed to talk loudly because they were talking —what can you do 
about that? You'll be hoarse at end of day, tired at end of day ... Then I'll present 
something as a strategy. (Mentor) 
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Finally, using web-based tools allowed mentors to complete some of the documentation 
requirements when they had a break between ECT meetings on site. In doing so, it freed their 
evenings up for other things. 


Documentation and logistical issues were time consuming and often took time away from 
mentoring. Both district coordinators and mentors commented on this challenge. Every year, 
mentors discussed the demands of their job and the extent to which they devoted more than 40 
hours a week to their work. In addition to documenting their own work, other areas that caused 
mentors to juggle their workload included scheduling, traveling to and conducting face-to-face 
visits, maintaining weekly communication with ECTs, and balancing their time with the needs 
of ECTs. UAF staff members acknowledged that travel and time constraints were challenges to 
implementation. 


I spend a lot of time on the paperwork, recording the face-to-face and observation 
information. That takes time. It’s like an IEP on steroids. (Mentor) 


The form and functionality of tools was sometimes questionable. One reason mentors 
incorporated district tools into their work was that they could not find ASMP tools that met the 
specific needs of the ECTs’ classroom, school, or district. This was most prominent when a 
district emphasized a particular approach to professional growth, as well as in some special 
education classrooms. 


Mentors and UAF staff members identified problems with the usability and accessibility of 
online tools. At the beginning of UGO, some tools were only paper-based, while others were 
available in both paper and electronic formats. While some mentors lamented that they needed 
more tools in an electronic format, others described difficulty in collecting classroom 
observation data only via computer. 


[It’s challenging] collecting data on what we agreed upon. I will notice that this is 
happening, and I will add tallies or draw a circle with arrows on who was talking to 
whom. I often improvise tools as needed, that is harder to do on a computer. (Mentor) 


This year is the first time I haven't attached all the tools that I complete with the teacher 
because of the issues with the tool suite and the amount of time to enter them. (Mentor) 


Documenting and reporting interactions with ECTs varied among mentors. Some mentors and 
UAF staff members acknowledged that some mentoring approaches or personal dispositions 
lent themselves to documenting and reporting mentor interactions with ECTs better than 
others. This included conversations that happened early in the relationship-building process, 
documentation that did not lend itself to monitoring implementation, and informal data 
collection. 
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My data collection, I spend a lot of time building relationships instead of collecting data. 
(Mentor) 


Finally, we identified one area in which UAF and UGOimplemented the key component of 
mentors use of Formative assessment tools differently from the model as planned (i.e., AK 
DEED’s implementation of ASMP). UGO mentors used a larger set of formative assessment 
tools, some that were district-developed and that supported school or district initiatives. 


Implementation Study Summary 


Evidence from the full spectrum of data sources indicates that ASMP successfully implemented 
UGO across all three years of systematically measured fidelity of implementation. The logic 
model accurately reflected implementation with effective program adaptations to an urban 
setting. UAF implemented most indicators with ideal fidelity. It implemented a few areas with 
adequate fidelity. 


District administrators selected mentors who had at least eight years of teaching experience in 
Alaska and fully released them from teaching responsibilities to devote their full attention to 
mentoring. However, the caseloads they carried were not always ideal. This was often due to the 
difficulty of districts not knowing in advance the number of ECT’s they would hire in a given 
year, which in turn made it difficult to hire the appropriate number of mentors to work with 
them. Usually, when caseloads were less than ideal, then full-time mentors had fewer than 12 
ECTs, rather than more than 15. Accordingly, some ECTs might have received more intense 
intervention as their mentor had additional time to spend with them. 


Mentors received the ideal amount of in-depth professional development by attending high- 
quality training events that provided them with background knowledge, training, and practice 
in engaging with ECTs and using the project’s reflective practice and formative assessment 
tools. In addition, there was time to network and engage with other mentors. These training 
events included Orientation, Academy week, Friday Forums, and Wrap Up. Mentors’ 
participation in coaching —and coaches’ documentation of those activities— was more variable. 
Mentors indicated their coach met with them during face-to-face visits with ECTs; however, the 
extent to which they communicated twice monthly and engaged in reflective practice was less 
uniform. The degree to which mentors reported needing/benefiting from this support, 
particularly in their second year of mentoring, also varied. 


Mentors conducted and documented weekly communications and engaged in reflective practice 
activities with their ECTs at consistently ideal levels. They met the criteria for adequate 
implementation in meeting with ECTs face-to-face and using formative assessment tools. 
Balancing caseloads, especially ECTs in varied locations and teaching positions, with ECT 
availability was a definite challenge. Mentor flexibility to use their best judgement to decide 
when and how to engage ECTs, at the best time and using the most appropriate tools, mitigated 
this challenge to some extent. Mentors found the tools valuable, especially as entry points for 
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difficult situations. This value outweighed requirements for completing documentation that 
mentors sometimes felt was menial work. Engaging in the minimum face-to-face time 
requirements and documenting use of formative assessment tools were two areas in which we 
found mentor differences, which we attribute, at least in part, to different mentoring styles. 


Across the project, ASMP staff members, district coordinators, and mentors acknowledged the 
importance of relationship building and ongoing communication and collaboration. These 
facilitated engaging in many aspects of work. On the administrative side, this included hiring 
mentors and identifying ECTs. On the mentoring side, this included developing as mentors, 
building trust with ECTs, and engaging in appropriate conversations with ECTs to move their 
practice forward. We can attribute many differences between implementation of the ASMP and 
UGO models to the fact that UGO is a district-level program, takes place in urban settings, and 
involves educators with well-defined teaching assignments but varied responsibilities. This is in 
contrast to the traditional ASMP model, which primarily takes place in rural schools in which 
ECTs may teach a variety of subjects and serve in numerous roles in the school. 
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Chapter 4. Intervention Study 


In this study we make a distinction between implementation and intervention. As described in 
the previous chapter, we use “implementation” to describe activities over which UAF had 
control, such as the actions described in the logic model (see Figure 1). Yet, we recognize that 
the intervention itself—mentoring — actually happens once mentors are out in the field working 
directly with ECTs. At that point, implementation is in the hands of the mentors. For this 
reason, the intervention study aimed to illuminate the “black box” of what really happened in 
UGO mentoring relationships. In this chapter we use terminology that is grounded in 
intervention science (Dunst, Trivett & Raab, 2013) to describe the interactions, activities, and 
actions mentors authentically engaged in with their ECTs. 


By examining verbal interactions between UGO-mentored ECTs and their mentors we can 
better describe the nature of the UGO mentoring intervention. And, because effective 
implementation (e.g., training and tools to use in an intervention such as mentoring) does not 
guarantee effective intervention (e.g., high-quality mentoring in practice) we also gain a deeper 
understanding of how UGO mentors applied their training to their actual work with ECTs 
(Dunst et al., 2013; Fixsen et al., 2005). 


Research Approach 


We framed the intervention study with this research question: What patterns in UGO mentor-ECT 
conversations are associated with improved ECTs’ instructional practice? One of the strengths of our 
overall research design is its breadth. We gathered, analyzed, and synthesized many different 
types of data about implementation, intervention, and impact. We also provided formative 
feedback from implementation and intervention data to UAF to support the project’s ongoing 
work. Based on this rich collection of data, we chose to take a mixed-methods approach to the 
intervention study. We used dyad conversation analyses to explore patterns in mentor-mentee 
conversations and their association with improved classroom practice. This included a 
combination of audio recordings of mentor-ECT post-observation conversations and 
instructional observation data. 


Intervention Analysis 


During their second year of mentoring, we asked UGO mentors to record four conversations 
with each of their second-year ECTs—two recordings each semester. These were to be post- 
observation debrief conversations. To ensure the recordings reflected the diversity and breadth 
of the work mentors did, we requested that mentors space recordings for a given ECT at least 
one month apart and that mentors capture a variety of different types of conversations. Mentors 
used audio recorders and either uploaded the audio files to a secure upload site, emailed files to 
us, or shared them via a Google Drive folder. 
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To take a deeper look at the UGO intervention itself, we conducted an analysis of the audio 
recordings and leveraged impact data to provide nuanced, actionable information to the project. 
The approach drew from CLASS data to inform our analysis. The CLASS data are based on 
scoring the instructional practice observations. Using results from the CLASS we examined two 
different groups of dyads: those with high rates of growth on the CLASS (referred to as Gliders) 
and those with low rates of growth (referred to as Sliders). By carefully studying audio 
recordings within these two groups, we aimed to detect contrasting patterns of how the dyads 
interacted, what actions mentors took to push ECTs’ practice, and what qualities ECT's exhibited 
that promoted or inhibited mentor work. Audio recordings were transcribed to facilitate coding 
using qualitative software. 


The analysis took a post-hoc, mixed-methods approach. We used ECTs’ spring-to-spring 
growth on the CLASS to group the dyads, audio recordings from the highest and lowest growth 
dyads, and an emergent qualitative analysis of the recordings to define — from the ground up— 
what mentors and ECTs did together that could contribute to the CLASS outcomes. Taken 
together, the CLASS dataand audio analysis provide a profile of mentoring experiences of UGO 
ECTs in Glider dyads and those in Slider dyads. 


Data Sources 

For this study, we drew on data from audio recordings of mentor-ECT post-observation 
conversations combined with scores from instructional observations using the CLASS. The 
study used multistage sampling with audio recording collected from all UGO-mentored ECTs 
in their second year of mentoring and CLASS data gathered from a stratified random sample of 
ECTs. Because we only have audio recordings for UGO-mentored ECTS, the analyses include 
UGO-mentored ECTs only. The full sample of ECTs with CLASS and audio data was 93. 


Methods and Participants 

We selected 10 ECTs (approximately 10% of the CLASS sample) who were at either end of the 
spectrum on their CLASS scores: five ECTs Gliders, who gained the most, and five ECT Sliders, 
who gained the least, based on differences in average fall Year 1 CLASS domain scores and 
average spring Year 2 CLASS domain scores. 


The CLASS uses a 7-point scale. A low score on the CLASS is 1-2, a midrange score is 3-5, and a 
high score is 6-7. Changes of a whole point are considered quite large. Gliders posted average 
gains ranging from 1.84 to 3.55 points. These are notable gains. The Sliders regressed on all 
three CLASS domains. They had the greatest decreases in total CLASS scores among the full 
sample of all those who were video recorded and had scored observations. Average decreases 
ranged from 1.98 to 3.85 points on the rating scale. 


Audio recordings of post-observation conversations between ECTs and mentors were 


transcribed and imported into Atlas-Ti 8.0, a software program designed for the management of 
qualitative data. Blind to whether recordings were from Glider or Slider dyads, we coded 
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mentor/mentee conversation data. We developed a coding scheme initially based on an earlier 
analysis of audio recordings, adding inductive coding to identify emergent differences and 
themes. A full description of the methodology is forthcoming in a separate paper (anticipated 
submission is the end of 2017). 


Findings 


Findings from this small exploratory study revealed pronounced differences between 
conversations of Glider and Slider dyads. With a limited sample of dyads, the purpose of our 
analyses was to investigate patterns in UGO mentor-ECT conversations that were associated 
with improved ECT instructional practice. Further research on these patterns is needed to better 
understand the many factors associated with Glider and Slider dyads. Three areas of interest 
emerged in examining differences between Glider and Slider dyads: dyad relationship 
dynamics, priority topics discussed in dyads, and types of mentoring activities in which dyads 
engaged. 


Dyad relationship dynamics and interaction. ECTs in Glider and Slider dyads were similar in 
terms of their relationship dynamics, as measured by mention of placement challenges, 
resistance to change, and attitude/disposition in general. Using the post-observation audio 
transcripts, researchers identified similar challenges in both dyad groups related to placement 
(e.g., fit in the school, grade level, content area). We detected comparable amounts of resistance 
to change expressed among Glider and Slider ECTs and similar attitudes (positive or negative) 
in both groups regarding teaching and mentoring. The similarity between the two groups in the 
area of dyad dynamics is important as it established that the Glider ECTs were not simply 
“easier” ECTs to work with or fortunate to be placed in better teaching situations. What we 
don’t know from the audio recordings is anything about the compatibility of the mentor-mentee 
relationship or the personal connections that mentors and ECTs may have felt. 


Glider dyads interacted with each other differently than Slider dyads. First, mentors and ECTs 
responded to each other more often. ECTs in Glider dyads were more engaged in conversation 
with their mentors (619 more instances of back-and-forth dialogue, .85 more instances per 
minute). Mentors in Glider dyads also affirmed or empathized with ECTs more frequently than 
Slider dyads (452 more instances, .64 more instances per minute). The Glider dyad 
conversations were longer, more focused on instruction, and more responsive to each other as 
compared with Slider dyad conversations. Glider dyads also had longer conversations than 
Slider dyads. On average, Glider conversations were about 5 minutes longer. In a 10-month 
academic year, this translates to 50 additional minutes of mentoring conversation. Fifty minutes 
is equivalent to almost two additional conversations per year, and as mentioned in the literature 
review, the amount of time spent mentoring is associated with better mentoring outcomes. 


Priority topics discussed in dyads. The post-observation conversations Glider dyads engaged in 


were qualitatively different from Slider dyads’ conversations in what they discussed. The 
Glider dyads focused their conversation more on instruction and students. Among Glider 
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dyads the topic of conversation was on instruction or practice in 606 more instances than the 
Slider dyads. That translates to .85 more instances per minute than the Slider dyads. Mentors in 
Slider dyads sometimes missed or minimized ECTs’ instructional challenges as they came up in 
conversation. Among Glider dyads discussion topics more often addressed student outcomes, 
work, or behavior. Glider dyads engaged in 521 more instances of discussing student outcomes, 
work, or behavior than Slider dyads (.76 more instances per minute than the Slider dyads). 
Glider dyads also problem solved together as mentor and ECT more often (252 more instances, 
.38 more instances per minute). Slider dyads sometimes had conversations that were only 
lightly facilitated by mentors and less often confronted problems of practice. Finally, Glider 
dyad conversations more frequently targeted ECTs’ successes and strengths. Mentors in Glider 
dyads focused on positive things going on in the classroom and with the ECTs’ instructional 
practice (265 more instances, .37 more instances per minute). 


Types of mentoring activities. Glider dyads were also qualitatively different from Slider dyads 
in the actions they took. Glider dyads more frequently focused on solving problems of practice 
and targeting successes and strengths of the ECT. Mentors in Glider dyads more often 
redirected conversations to the positive, especially with regard to students. Glider dyad 
conversations picked up on challenges and addressed them, collaboratively generating next 
steps or new approaches for ECTs to try. Mentors in Glider dyads directly facilitated 
conversations when needed, not letting discussions get derailed by extraneous conversations or 
avoidance of difficult topics. 


Intervention Study Summary 


The intervention study examined what actually happened as mentors and ECTs interacted. The 
analyses of CLASS and audio data identified discernible differences in UGO dyad conversation 
patterns associated with improved ECT instructional practice. Glider dyads were qualitatively 
different from Slider dyadsin the way they interacted with each other, the topics they 
discussed, and the actions they took. They had longer conversations, focused more explicitly on 
instruction and students, responded to each other more often, and engaged as peers more 
frequently than the Slider dyads. 


Findings from this small exploratory study offer a window into the black box that is UGO 
intervention. Although not detailed in this overview of the study, the ASMP cycle of mentoring 
is apparent in the interaction patterns among Glider dyads. Future research is needed to 
develop an understanding of the conversation patterns that emerged among dyads and how 
they may be linked to instructional practice as measured by tools such as the CLASS. 
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Chapter 5. Impact Study 


Research Questions 


The main purpose of this research was to estimate the impact of ECT participation in UGO. 
Three research questions guided the impact study: 
1 What is the impact of ECTs’ participation in UGO on remaining in the teaching 
profession in Alaska? (RQ1) 
2 What is the impact of ECTs’ participation in UGO on their instructional practices? (RQ2) 


What is the impact of ECTs’ participation in UGO on the reading, writing, and 
mathematics achievement of their students? (RQ3) 


Methods 


Intervention and BAU Conditions 

ECTs within each cohort were randomly assigned to intervention (UGO) or control (business as 
usual, or BAU) conditions within each district. ECTs in the intervention condition received two 
years of mentoring through UAF and did not receive other formal mentoring offered in their 
schools or districts. ECTs assigned to the control condition received no UGO mentoring, instead 
they receive BAU mentoring, defined as mentoring typically provided to new teachers in the 
absence of UGO. Formal BAU mentoring varied in quality and intensity at the two districts that 
provided it, and there was no formal BAU mentoring in the other three districts. 


Analytic Methods and Models, Statistical Adjustments, and Missing Data 

In the following section, we discus analytic models by research question. All models include 
cohort-by-district fixed effects that reflect the random assignment of teachers within blocks 
formed by cohort and district. 


RQ 1: What is the impact of ECTs’ participation in UGO on remaining in the teaching 
profession in Alaska? 


Confirmatory outcome: 
1. Whether or not an ECT participant remained a teacher in Alaska in the third year of 
teaching. 
Exploratory outcome: 
2. Whether or not a Cohort 1 ECT remained a teacher in Alaska in the fourth and fifth year 


of teaching. 


The outcomes are binary variables. 
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Analytic models: 
For each of the two outcomes (retained in Year 4 and retained in Year 5), the logistic model at 
the teacher level is: 


x 
logs ae = By + fP,2Cohort dummy variables + B3_¢District dummy variables 


+ B7_4,Cohort X District interaction variables + B,,Treatment indicator 


In each model, the coefficient for the treatment indicator estimates the impact of participating in 
UGO on retention. 


Analyses 

We conducted one confirmatory and three exploratory analyses to examine the impact of ECTs’ 
participation in UGO on remaining in the teaching profession in Alaska. Our confirmatory 
analysis compared the retention rates of UGOand BAU ECTs at the beginning of their third 
year in the teaching profession, after UGO ECTs received two years of UGO mentoring and 
BAU ECTs received two years of the traditional BAU program. For our exploratory analyses, 
we calculated retention for Cohort 1 ECTs only, in their third, fourth, and fifth year of teaching. 


Attrition 

We had low attrition (for details see Consort Charts in Appendix D). Some ECTs were missing 
outcome data because we did not have their state identification number, which were missing 
from the files we received from AK DEED (including the two years they were teaching in the 
study). 


Baseline equivalence 

Although attrition was below the level required to establish baseline equivalence, we did 
compare the baseline equivalence of ECI's based on the WWC protocol acceptable measure for 
teacher retention outcomes of teaching experience (U.S. Department of Education, 2016). All 
treatment and control ECTs began the study in their first year as classroom teachers (i.e., no 
difference between groups at baseline). Additionally, research suggests two related but separate 
correlations to retention: Alaska teachers who earn their degree in Alaska (Hirshberg & Hill, 
2013) have higher retention than those who do not earn their degree in Alaska, and Alaska 
teachers who do not relocate to Alaska for work have higher retention than those who do 
relocate to Alaska for work (Boyer, 2012, p.49). These descriptives regarding whether ECTs 
earned their degree in Alaska and relocated to Alaska for work are displayed in Table 5 in 
Chapter 1. At baseline, there were no differences (effect sizes of 0.00 and -0.07, respectively). 


RQ 2: What is the impact of ECTs’ participation in UGO on their instructional practices? 
Outcomes: 


1. CLASS scores on the domain of Emotional Support 
2. CLASS scores on the domain of Classroom Organization 
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3. CLASS scores on the domain of Instructional Support 


Instructional practice scores were obtained from ratings of video recordings of teachers in their 
classrooms, which were assigned by raters trained in the CLASS observation system. Outcome 


variables are means of the scores on the components that form each of the three domains 
covered by the CLASS. 


Analytic models: 

As before, in each model the coefficient for the treatment indicator estimates the impact of 
participating in ASMP on teachers’ instructional practices. Results are produced separately for 
elementary and secondary levels. 


Vi = Po t+ Bx; + B3,4Cohort dummy variables + B;_,District dummy variables 
+ Bo_4,Cohort X District interaction variables + B,,Treatment indicator + e; 


Where: 

yi= the outcome value on emotional support, classroom organization, or instructional support 
for teacher i 

Bo= the intercept 

xi1= value of baseline measure on emotional support, classroom organization, or instructional 
support, for teacheri 

B1, 2, bk = regression coefficients 

Bis-estimate of impact of ASMP on on emotional support, classroom organization, or 
instructional support 

ei-error term for teacher i 


Analyses 

We conducted three confirmatory analyses to examine the impact of ECTs’ participation in 
UGO on their instructional practices. Analyses used results of instructional observations as 
measured by the CLASS. For our confirmatory analyses, for the three CLASS dimensions 
(emotional support, classroom organization, and instructional support), we conducted pooled 
analyses of ECT’s scores at the end of their second year of teaching. We included all Cohort 2 
and Cohort 3 ECTs who were randomly assigned to instructional recordings prior to their first 
year of teaching. 


Attrition 

We had a moderate but acceptable attrition rate (27.6%). Attrition was below the level required 
to establish baseline equivalence; however, these statistics are shown in table 15. To further 
increase the precision of the analytic model, we included the baseline as a covariate. We were 
missing outcome data for ECTs who dropped from the study prior to video recording in spring 
of their second year of teaching. We also had missing outcome data for those ECTs who were 
recorded in their first year of teaching but assigned to classrooms in their second year of 
teaching in which we made a priori decisions to exclude from video recording. Furthermore, all 
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districts in Year 2 excluded special education teachers from video recording. We also excluded 
physical education teachers from video recording because it was difficult for teachers to exclude 
from the lessons the students who did not have permission to be video recorded. Other reasons 
for not video recording ECTs included: students who were involved in the juvenile justice 
system or were dropouts, human error, ECTs who declined to be video recorded after agreeing 
to be recorded, and illness. One special education teacher in the UGO condition assisted 
students in the classroom of another video recorded ECT in the BAU condition. Since we could 
not attribute CLASS scores to the “pushed in” teacher, we excluded the teacher from the 
analysis. 


RQ 3: What is the impact of ECTs’ participation in UGO on the reading, writing, and 
mathematics achievement of their students? 


Outcomes: 

The outcome variables for RQ3 are student achievement scores obtained from AK DEED. 
Reading, writing, and mathematics scores were collected for grades 4—6. Mathematics scores 
were collected for grades 7-10. These data were available for each study year. Baseline scores 
were collected annually from the summer of 2012 to the summer of 2015. Note that the baseline 
scores in reading and writing were collected for grades 3-5. The baseline score in mathematics 
was collected for grades 3-9. 


Alaska changed its state tests from the Standards Based Assessment (SBA) to Alaska Measures 
of Progress (AMP) during the course of this study. The SBA was used in the spring 2014 for the 
last time. As a consequence of this change, depending on the cohort, the baseline and the 
outcome scores could both be in SBA (e.g., Cohort 1, Year 2 students) or could be a mixture of 
SBA and AMP (e.g., Cohort 2 , Year 2 students). This complication forced us to take additional 
steps to analyze student data pooled across all three cohorts. 


SBA and AMP measure student competency in different content areas. Alaska did not equate 
them. Furthermore, state tests typically measure different content areas for different grade 
levels and are generally not vertically equated. As a consequence, we established concordance 
between SBA and AMP and also across different grades through standardization, then 
performed a pooled impact analysis. 


Specifically, the pooled analysis process involved the following steps: 
1. Convert student-level scores on state tests (SBA or AMP) into z-scores for each test year, 
using state-level means and standard deviations within subjects and grades. 
2. Estimate the impact separately for each cohort and grade. (In other words, for each 
cohort-by-grade combination.) 
3. Pool the impact estimate across cohorts and across grades, using the precision-weighting 
method. 
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For example, the pooling process for estimating the overall impact of “ECT’s two-year 
participation in UGO on the reading performance of primary grade students” involved: 

1. Converting state reading test scores into z-scores, for each year and for each grade. 
There will be a total of 18 (3 x 3 x 2) standardizations, as the reading test scores will 
consist of data from three cohorts of students in grades 4—6, each with baseline and 
outcome scores. 

2. Estimate the impact separately for each cohort and for each grade, which will result in 
nine (3 x 3) separate impact estimates. 

3. Aggregate the nine impact estimates using the precision weights, which will be the 
inverse of the squared standard error associated with each impact estimate. 


The second and third steps were combined by including cohort and grade as design variables in 
the impact analysis model for RQ3 to reflect pooling of test scores. The model includes the 
treatment indicator (UGO vs. BAU). Covariates include the baseline test score and student 
demographic variables such as gender, race/ethnicity, and FRL status. In addition, random 
assignment of teachers within blocks formed by cohort and district is reflected in the model by 
the indicator variables representing the cohort-by-district interaction. 


The impact analysis utilizes two-level hierarchical linear modeling to account for the nesting of 
students within teachers. 


Joiners. The impact study assigned ECTs to conditions randomly. However, it is important to 
understand the implications for randomization at the student level arising out of when students 
were “attached” to ECTs relative to the timing of randomization of teachers. For Year 1, 
students were randomly assigned if they were “attached” to ECTs at the time teachers were 
randomized to conditions. On the other hand, in Year 2 students cannot be considered to be 
randomized because that summer, principals knew the treatment status of ECTs and could have 
steered students to teachers accordingly. Asa result, Year 2 students would be considered study 
“joiners”. 


Contrasts. The impact study for student achievement includes a total of 10 contrasts from our 
original design plan. All are between students in the classroom of the UGO ECTs versus those 
in the classroom of BAU ECTs. Six of the contrasts estimated the impact at the end of the first 
year of teaching, by pooling the end-of-first-year data from the classrooms of ECTs in Cohorts 
1-3 (pooling the 2012-13 data, 2013-14 data, and 2014-15 data). These six contrasts are for 
exploratory impact analyses. The other four contrasts estimated the impact at the end of the 
second year of teaching by pooling the end-of-second year data from the classrooms of ECTs in 
Cohort 1 and 2 within each educational level (pooling the 2013-14 data and 2014-15 data). These 
four contrasts are for confirmatory analyses. (Alaska did not administer student assessments in 
2015-16.) 


Additionally, the impact study for student achievement included exploratory analyses not 
originally written into the design plan. The additional exploratory analyses for primary reading 
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and secondary mathematics estimated the impact at the end of the first year of teaching, as 
described above. All contrasts are shown in Table 13. 


Table 13. Contrasts per design plan and additional exploratory contrasts 


Subject Level Year Role 
1 Reading Primary 2 Confirmatory 
2 Writing Primary 2 Confirmatory 
3 Math Primary 2 Confirmatory 
4 Math Secondary 2 Confirmatory 
5 Reading" Primary 1 Exploratory 
6 Reading? Primary 1 Exploratory 
7 Writing Primary 1 Exploratory 
8 Math Primary 1 Exploratory 
9 Math Secondary 1 Exploratory 
10 Math? Secondary 1 Exploratory (race by treatment interaction)' 


1 This contrastis for the full sample. 

? This contrast estimates the impact of first-year, UGO-mentored ECTs on primary students’ reading achievement as 
compared to students of first-year BAU ECTs who had no formal mentor. Contrastis exploratory because it was not 
part ofour original design plan. 

3 Examined separately for white, Hispanic, American Indian, Alaska Native, and multiracial students 

Note: The four comparisons for the confirmatory analysis address different combinations of subjects and grade levels. 
As a result, p-values for the impact estimates were not adjusted for multiple-comparisons. 


The analytic model for student achievement is a student-within-teacher mixed model with the 
same general form for each contrast: 


Level 1 (student) model: 


Vy = Boy + Bij (Baseline;, ) + Bo-3; (Cohort indicators; ;) 
+ B4~10; (Race — ethnicity indicators; ;) + Byy-14; (District indicators; ;) 
+ Bi6j (Economic disadvantage ;;) + By7; (Male; ;) + Big-20; (Grade indicators;;) 
+ Boi; (Attendance; ) + Bo2-29; (Cohort by district indicators; ;) 
+ B30; (Treatment; ) + 1; 


Level 2 (teacher) model: 


Boj = Yoo + Uo; 
Bij = Yio + Uy; 


The mixed model form is: 


Vij = Yoo + Y10 (Baseline;; ) + ¥2-30 (Cohort indicators; ;) 
+ Y¥4-10,0 (Race — ethnicity indicators;, 7) + ¥41-14,0 (D istrict indicators; i) 
+ V16,0 (Economic disadvantage j;) +717,0 (Male; ;) + ¥18-20,0 (Grade indicators; ;) 
+ ¥21,0 (Attendance; ) + ¥22-29,0 (Cohort by district indicators; ;) 
+ ¥30,0 (Treatment; ) + Ug + Nj 
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For economy, the formulation above ignores unique effects on each student-level slope due to 
teachers. 


The major coefficients of interest are: 

Yij =the outcome for student 7 within teacher j 

30= the estimated impact of the treatment on student i within teacher j 
Uoj= unique increment to the intercept associated with teacher j 

rj = residual associated with student i within teacher j 


Analyses 

We conducted four confirmatory and six exploratory analyses to examine the impact of ECTs’ 
participation on the reading, writing, and mathematics achievement of their students. For our 
confirmatory analyses, we analyzed state assessment scores from primary reading, writing, and 
math and secondary math scores of students of ECTs at the end of their second year of teaching. 


Attrition 

Attrition for primary reading, writing, and math and secondary math students of ECTs at the 
end of their second year of teaching was low (10.0%, 10.9%, 11.1%, and 12.4%, respectively). 
Attrition was below the level required to establish baseline equivalence; however, these 
statistics are shown in table 18. To further increase the precision of the analytic model, we 
included the baseline as a covariate. Students were dropped from the analyses if they were 
missing outcome data (students who did not participate in the state assessments) or covariate 
data. For exploratory analyses we analyzed the reading, writing, and mathematics state 
assessment scores for the students of ECTs at the end of their first year of teaching (after the first 
year of exposure to the treatment). We also analyzed data for students of different 
race/ethnicities and in districts without formal mentoring programs. 


Findings 


UGO ECTs Were Retained as Teachers in Alaska Public Schools at Higher Rates Than 
BAU ECTs, but Not at Statistically Significant Levels 

UGO ECTs were retained in teaching at a higher rate than the BAU group. For all ECTs in their 
third year of teaching, we found an 80.5 percent retention rate for UGO ECTs compared to a 
76.6 percent retention rate for BAU ECTs. While this finding indicates the UGO group had a 
higher retention rate than the BAU group, it represents a non-statistically significant difference 
(at the level of p < 0.05) with an effect size of 0.16. Table 14 displays the post-intervention 
statistics for the full sample of ECTs assigned to the UGO and BAU conditions. 


Because we had three years of retention data for Cohort 1 ECTs, we were able to analyze their 
retention over a longer period (fall 2015, 2016, and 2017). Our analyses show that in all three 
years, UGO ECTs had higher, but non-statistically significant retention rates (80.5%, 72.7%, and 
68.8%) compared to BAU ECTs (74.3%, 62.9%, and 65.7%). Compared to ECTs third year of 
teaching, retention was lower for both groups in their fourth year of teaching. In their fifth year 
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of teaching UGO ECTs still had higher retention compared to BAU ECTs, but BAU ECTs had 
higher retention than in their fourth year of teaching. By their fifth year in the teaching 
profession, about two-thirds of UGO and BAU ECTs remained in the profession. The effect size 
for Cohort 1 ECTs in their fourth year of teaching (0.274) suggests differences that may be 
substantively important but not statistically significant, based on guidelines established by the 
What Works Clearinghouse (NWWC) (U.S. Department of Education, 2014). 


Table 14. Estimated impact on teacher retention 


UGO group BAU group Estimated effects 


Impact | p-value 
Mean Mean estimate 
(standard (standard || (standard 
Outcome measures deviation)' deviation) error) 


Retention Year 3 0.805 0.766 1.294 
All cohorts, pooled eras oe ieacya el nua ede 
0 . 


Retention Year 3 0.805 0.743 1.431 

(Cohort 1 only) (0.399) i (0.440 (0.568) | 9-367 | 0.216 

Retention Year 4 0.727 0.629 1.576 

(Cohort 1 only) vas (0.448) (0.487) || (0.561) | 2201 | 0.274 

Retention Year 5 0.688 0.657 1.152 

Cohort 1 onl 0.466 0.478 O-406)0.)|. Poe” 1), 0088 
1 Teacher level standard deviations calculated from sample shown in table. 


2 Cox index. 
3Confirmatory contrasts are included in shaded rows. 


No Statistically Significant Differences between UGO and BAU ECTs Were Found on 
Instructional Practice as Measured by the CLASS 


For the pooled sample of ECTs, the CLASS domain scores of emotional support, classroom 
organization, and instructional support for ECTs in the fall of their first year of teaching were 
higher for the BAU group than the UGO group. Scores fell in the midrange (3-5), and trends 
mirrored national trends with higher scores in the emotional support and classroom 
organization domains and lower scores in the instructional support domain. Effect sizes for the 
pooled sample ranged from -0.16 to -0.26. In regard to confirmatory contrasts, we established 
baseline equivalence for the pooled K-10 ECT sample for the classroom organization and 
instructional support domains, but not for the emotional support domain. Table 15 displays the 
pre-intervention, baseline statistics for the impact analysis sample for each contrast. 


Table 15. Pre-intervention sample sizes and characteristics for the analytic sample of ECTs in 
instructional practice analysis 
Baseline UGO group BAU group Effect 


measures ; Sample : Sample size® 
(CLASS) 


Number Number 
Number in Un- Number in Un- 


randomly impact adjusted randomly impact adjusted 
assigned | estimate mean assigned | estimate Mean 


0.658 
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Baseline BAU group Effect 


| _UGOgroup———sd 
measures : Sample ' Sample size® 
(Sass) 


Number Number 
Number in Un- Number in Un- 


randomly impact adjusted randomly impact adjusted 
assigned | estimate mean assigned | estimate Mean 


0.566 


1CLASS as the baseline measure. ES, emotional support domain; CO, classroom organization domain; |S, 
instructional support domain. 

2S.D. is teacher level standard deviations calculated from unadjusted statistics from impact estimate sample. 
3 Hedges G. 

4 Confirmatory contrasts are included in shaded rows. 


Post-intervention statistics show that BAU ECTs at the end of both their first and second years 
of teaching obtained higher scores —but not at statistically significant levels—on the three 
CLASS domains, with effect sizes that ranged from -0.32 to -0.14. In the domains of emotional 
support and classroom organization, differences were larger at the end of Year 1 than Year 2. 
The difference between the instructional support scores of BAU and UGO ECTs was larger in 
Year 2 than in Year 1. Again, most scores fell in the midrange (3-5). The classroom organization 
scores of the BAU group in Years 1 and 2 fell in the high range (6 or 7). Trends for the Alaska 
teachers continued to mirror national trends with higher scores in the emotional support and 
classroom organization domains and lower scores in the instructional support domain. Table 16 
displays the post-intervention statistics for the impact analysis samples, which were less than 
what was expected. No statistically significant differences were found. 


Table 16. Post-intervention outcomes for the analytic sample and estimated effects of UGO on 
ECTs’ instructional practices 


Baseline || |=UGOgroup _—i Estimated effects 


measures Impact 
(CLASS)' estimate p-value 
Standard Standard || (standard | (degrees of 
deviation? deviation error) freedom) 
0.297 -0.283 
@isn | dite) | | 
0.139 11, 85 
0.109 11, 85 
-0.018 0.873 -0.163 
Fora | dies | | 
0.118 11, 85 
0.113 11, 85 
1ES, emotional support domain; CO, classroom organization domain; IS, instructional support domain. 
? Teacher level standard deviations calculated from unadjusted statistics from impact estimate sample. 


3 Hedges G. 
4 Shaded rows identify confirmatory contrasts. 
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ECTs Report that UGO Mentoring Positively Impacted their Instructional Practice 
In the annual survey, we asked ECTs to report the extent to which participating in different 


activities and discussing specific topics with their mentor was influential on their instructional 
practice. UGO ECTs reported a statistically significant greater influence (p < 0.05) in regard to 
face-to-face visits, distance communication, brainstorming, observing instruction, collecting 
data, goal setting, documenting their work together, discussing observations, and discussing 


equity (Table 17). 


Table 17. Mentoring activities with significantly more influence on teaching practice* 
Percentage (n) 


Mentoring activity Group 


Brainstorm with ECT ways to BAU 
approach a challenge witha 


student or class UGO 

BAU 
Collect classroom data 

UGO 

; aa BAU 

Discuss equity issues 

UGO 
Discuss observations of ECT's BAU 
instruction and/or data that were 
gathered UGO 
Document ECT/ BAU 
mentor’s work together UGO 
Engage in face-to-face BAU 
communication UGO 


Engage with ECT ina goal-setting BAU 


process UGO 
Engage with ECT in distance BAU 
communication UGO 

BAU 


Observe ECT’s instruction 
UGO 


Not at all/ 


Hardly at all 


2.4% (8) 


3.3% (17) 


Some 
25.0% (85) 
13.6% (71) 
50.0% (72) 

31.9% (159) 
43.9% (127) 
29.6% (149) 
48.5% (95) 
20.1% (88) 
41.1% (97) 
35.5% (179) 
35.2% (123) 
25.0% (130) 
39.6% (97) 
26.6% (138) 
47.0% (141) 
47.4% (237) 
44.7% (72) 
26.2% (135) 


Quite a bit/a 
great amount 


72.7% (247) 
85.1% (446) 
39.6% (57) 
65.7% (327) 
52.3% (151) 
67.3% (339) 
45.4% (89) 
78.4% (344) 
39.4% (93) 


46.6% (75) 
70.5% (363) 


*Note: Mentoring activities and discussion topics that UGO ECTs reported as having statistically, significantly greater 
influence (p< 0.05 using chi-squared) ontheir practice than BAU ECTs. 


UGO ECTs were also more likely to indicate their mentoring was, in general, impactful. 
Compared to BAU ECTs, UGOECTs were more likely to agree their work with their mentor 
was beneficial to their teaching (79% UGO versus 67% BAU strongly agreed). Additionally, they 
attributed larger proportions of their success to their mentor (66% UGO versus 54% BAU). 
UGO-mentored ECTs also indicated having a mentor (formal or informal) enhanced their 
teaching at higher rates than BAU ECTs (94% UGO versus 49% BAU). Complete data tables 


may be found in Appendix A. 
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These results provide ECTs’ perspective on the positive impact of having a mentor. Classroom 
observation identified no statistically-significant differences in instructional practices between 
UGO ECTs and BAU ECT. However, across mentoring activities, UGO ECTs believed their 
mentors had an influence on their teaching at rates that were statistically significantly higher 
than the BAU ECTs. 


Student Achievement Was Generally Higher for Students of UGO-Mentored ECTs, and 
the Achievement of Some Student Groups Was Positively Impacted at Statistically 
Significant Levels 

The baseline scores of primary students (grades 4-6) of UGO ECTs on the reading, writing, and 
mathematics state assessments were typically lower than the scores of students of BAU ECTs 
(effect sizes ranged from 0.01 to 0.14). The baseline scores of secondary students (grades 7-10) of 
UGO ECTs on the mathematics state assessment were higher than those of BAU ECTs (effect 
sizes ranged from 0.31 to 0.48). We established baseline equivalence for the primary samples, 
but not the secondary samples. Table 18 displays the pre-intervention baseline statistics for the 
impact analysis sample for each contrast. Confirmatory contrasts are shaded in the table. 


Table 18. Pre -intervention sample sizes and characteristics for the analytic sample in student 
achievement analyses 
Baseline UGO group BAU group Effect 


4 3 
measures Sample Sample size 
Sample sizes P Sample sizes P 
characteristics characteristics 


eure ae 
Number 
Number of vide of eek 
students in Un- students in Un- 
randomly analytic | adjusted randomly | analytic | adjusted 
assigned sample Mean assigned sample Mean 


Reading -0.051 


Writing ae ice ee 
Writing PY2 
Math PY 1 
Math PY2 
Math SY 1 
| 809 | = 696__—| -0.087_| 0.994 | 


Reading 

PY1 

Mili 0.238 | 0.868 0.216 | 0.894 | 0.025 
mentored 

teacher 

Math SY1 

eae —— i i ES ia 0200 oe 
students 

Math SY1 

SoC eco 
students 
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Baseline BAU group 


measures' Sample Sample 
Sample sizes Sample sizes 
characteristics P characteristics 


see 
Number 
Number of Pe of students 
students in Un- students in 
randomly analytic | adjusted randomly | analytic | adjusted 
assigned assigned sample Mean 


Math SY 1 
(Alaska 146 -0.615 | 0.825 
Math SY1 
(American : 15 13 -0.534 | 0.807 
eli a : 139 -0.372 | 0.888 


1P, primary; S, secondary, Y1, spring - first year of teaching; Y2 spring - second year of teaching. 

2 Student level standard deviations calculated from unadjusted zscores of the impact estimate sample. 
3Hedges G. 

4 Confirmatory contrasts are included in shaded rows. 


Sample sizes at the cluster and individual levels may be found for all contrasts in the contrast 
table (Appendix D). 


Most student groups with UGO-mentored ECTs scored higher on the state assessment than 
student groups with BAU teachers. (The exceptions are primary reading students at the end of 
their ECT’s second year of teaching and primary writing students at the end of their ECT’s first 
year of teaching.) All but one impact estimate was positive (the exception being students of 
first-year secondary math UGO ECTs), but no differences were statistically significant. Table 19 
shows the statistics for the impact analysis samples. 


Students of both first- and second-year UGO ECTs improved their performance on the state 
mathematics assessment. The effect size of the difference between the scores of students with 
UGO-mentored teachers and students of BAU teachers for first-year secondary mathematics 
ECTs (0.25) suggests substantively important but not statistically significant differences, based 
on guidelines established by the WWC (U.S. Department of Education, 2014). 


66 Education Northwest 


Table 19. Post-intervention outcomes for the analytic sample and estimated effects on student 
achievement 


Outcome Intervention group Comparison group Estimated effects 


measures! Impact 
estimate 
Standard Standard (standard Effect 
deviation” deviation error) p-value size® 
| 0.962 | 0.324 0.069 
0.093 0.877 a Say 
0.053 0.945 pas 
0.018 0.741 0.119 
0.053 0.528 0.072 
-0.019 0.642 0.247 
0.005 0.954 0.174 


Reading 

PY1 

(mentored 0.147 

eee 0.279 0.113 0.931 (0.063) 0.019 0.176 
mentored 

teacher) 


Math SY1 0.254 
(Caucasian |} 0.129 0.932 -0.134 0.862 (0.096) 0.008 0.292 
students 7 
Math SY1 0.224 
(Hispanic -0.365 0.792 -0.573 0.764 0.108 0.037 0.265 
students (0. ) 
Math SY1 Aen 
(American iF z : 

0.170 0.668 0.782 1.034 (0.231) 0.052 0.659 
Math SY1 
(Alaska 0.236 
Native -0.300 0.890 -0.547 0.833 (0.107) 0.028 0.286 
students 


Math SY1 

(Students 0.243 

of two or -.136 -0.381 0.875 (0. 110) 0.028 0.288 
more ; 

races 


1P, primary; S, secondary, Y1, spring - first year of teaching; Y2 spring - second yer of teaching. 

? Student level standard deviations calculated from sample shownin previous table (unit of analysis) and z-scored 
prior to analysis. 

3 Hedges G 

4 Confirmato ry contrasts are includedin shaded rows. 
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In exploratory analyses, we found positive effects on a diverse set of students in mathematics 
achievement after their teachers had one year of UGO mentoring. As shown in Tables 18 and 19, 
we conducted exploratory analyses to estimate the impact of UGO mentoring on students of 
varied races/ethnicities. The secondary white (Caucasian), Hispanic, Alaska Native, and 
students of two or more races (not Hispanic) of first-year ECTs obtained significantly higher 
scores on the state math assessment than students with BAU teachers ( p values ranged from 
0.008 to 0.037). American Indian students with first-year UGO-mentored ECTs also achieved 
higher scores on the state mathematics assessment than American Indian students with BAU 
teachers (p =0.052). 


In another exploratory analysis, we found primary students’ reading achievement was 
positively impacted by first-year UGOECTs as compared with BAU ECTs who had no formal 
district mentor. We conducted an exploratory analysis to determine whether UGO mentoring 
had any impact on the students of UGO ECTs in districts that did not have formal mentoring 
programs (i.e., Fairbanks, Mat-Su, and Sitka). The primary students of first-year ECTs obtained 
significantly higher scores on the state reading assessment than students of BAU teachers 

(p = 0.019) with an effect size of 0.176. 


Impact Study Summary 


Teacher retention in the third year of teaching in Alaska was higher for the UGO ECTs than the 
BAU group. However, differences were not statistically significant. The effect size of 0.274 for 
Cohort 1 ECTs retained in their fourth year of teaching suggests positive effects on UGOECTs 
that may be substantively important. 


There was no statistically significant effect of ECTs’ participation in UGO compared with ECTs’ 
participation in BAU on teacher instructional practice as measured on the three domains of the 
CLASS—emotional support, classroom organization, and instructional support. CLASS scores 
were higher for the BAU ECTs compared to UGO ECTs, but not significantly so. 


There was no statistically significant effect of ECTs’ participation in UGO compared with ECTs’ 
participation in BAU on primary students’ academic achievement in reading, writing, or math 
and secondary students’ academic achievement in math. State assessment scores were higher 
for the primary students of UGO ECTs in reading and math and for secondary students in 
math compared to BAU ECTs. State assessment scores were higher for the primary students 
of BAU ECTs in writing compared to UGO ECTs. However, these differences were not 
statistically significant. 


We found positive effects on a diverse set of students in mathematics achievement after their 
teachers had one year of UGO mentoring. The secondary (grades 7-10) white (Caucasian) 
students, Hispanic students, Alaska Native students, and students of two or more races (not 
Hispanic) of first-year UGO ECTs obtained significantly higher scores on the state mathematics 
assessment than students with BAU teachers. These differences were statistically significant (p 
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values ranged from 0.008 to 0.037). American Indian students of first-year UGO ECTs also 
achieved higher scores on the state mathematics assessment than American Indian students 
with BAU teachers (p =0.052). 


Primary students’ reading achievement was positively impacted by first-year UGOECTs as 
compared with BAU ECTs who had no formal district mentor. The primary (grades 3-6) 
students of first-year UGO ECTs obtained statistically significantly higher scores on the state 
reading assessment than students of BAU ECTs without a formal mentor (p =0.019). 
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Chapter 6. Summary 


Implementation Study Findings 


Using the logic model as a basis for measuring implementation fidelity, we created a fidelity of 
implementation (FOI) matrix. The FOI matrix includes stated expectations from UAF 
(indicators) for implementation of UGO’s four key components: mentor recruitment and 
assignment, mentor participation in professional development, mentor interactions with their 
ECTs, and mentor use of formative assessment tools. In addition to the key components and 
indicators, the matrix also identifies three levels of implementation: low, adequate, and ideal. 


Results of the implementation study indicate that in most areas the UGO implementation of the 
ASMP model was at the ideal level, and evidence from the full spectrum of data sources 
indicates that ASMP successfully implemented UGO across all three years of systematically 
measured fidelity of implementation. The logic model accurately reflected implementation with 
effective program adaptations to an urban setting. 


For indicators with less than ideal fidelity, several situations may have contributed to this, 
including challenges related to estimating the number of new teachers to be hired each year, 
which influenced how many mentors would be needed; differences among mentors in 
recording activities (e.g., coaching and use of formative assessment tools); and differences in the 
availability of ECTs in urban versus typical ASMP (rural) settings, which may have influenced 
the amount of time mentors spent with their ECTs. 


The initial teaching experiences of UGO and BAU ECTs was substantially different in terms of 
the activities they engaged in at their school or district and their mentoring. As compared with 
BAU ECTs, these differences may have contributed to UGO ECTs spending more time with 
their mentor, being more trusting of their mentor, and engaging in work with their mentor that 
impacted their instructional practice. 


The survey analysis revealed significant differences between the UGO and BAU mentoring 
interventions. UGO and BAU ECTs thought about their mentor’s roles differently. UGO ECTs 
were more likely than BAU ECTs to think of their mentor as an expert guide, role model, 
advocate, and therapist/counselor. Whereas larger proportions of BAU than UGO ECTs 
considered their mentor a colleague, which is reasonable considering they also reported that 
their mentors were typically colleagues in their school. UGO ECTs met with their mentors less 
frequently but for longer periods than BAU ECTs. Possibly, to make up for less frequent face-to- 
face meetings, and compared to BAU ECTs, UGO ECTs had more frequent distance 
communication (e.g., telephone, email, text) with their mentors—75 percent of UGO ECTs 
communicated via these means daily or weekly versus 49 percent of BAU ECTs who did so. 
BAU ECTs were more likely to be mentored informally face-to-face. 
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Several factors facilitated implementation including focus on relationship building and ongoing 
communication and collaboration with project leaders, coordinators, mentors, and ECTs; high 
quality mentor professional development with required attendance, and training that effectively 
prepared mentors to work with ECTs. UGO mentors were respected as professionals, 
empowered to use their professional judgment and able to exercise flexibility and mentors 
found the ASMP documentation and formative assessment tools valuable. 


Some conditions challenged implementation including a variety of logistical issues related to 
identifying and working with ECTs (e.g., estimating the number of mentors needed each year, 
balancing caseloads and travel, and organizational policies on hiring/contracting); the need for 
differentiated training and coaching to provide engaging content and professional development 
formats to mentors at different stages of mentoring and experience levels; and an initially rocky 
rollout of mentor tools online which inhibited use during the early rollout period. Mentors also 
had to balance relationship building with ECTs and their own mentoring style with tool use. 
Not all mentoring activities lent themselves to using tools and not all mentors were inclined to 
document every activity with their ECTs. 


There was little variation in implementation across mentors. However, the UGO model varied 
from the ASMP model in some ways that stemmed from the fact that UGO was essentially a 
district initiative implemented in five districts, versus a state initiative implemented in many 
districts (that are often in rural, isolated villages or communities). This affected hiring and 
contracting, mentor access to support, and the amount of district-specific support ECTs received 
from their mentors. 


There were some important differences between UGO and BAU conditions that distinguish the 
UGO from BAU groups, including: 

e Inthe two districts with formal mentoring programs, ASMP was programmatically 
different. There were few similarities in ASMP’s key components across the district- 
sponsored mentoring programs. 

e BAU ECTs were more likely than UGO ECTs to receive or engage in some school or 
district support activities that commonly involved work with other colleagues. 

e The mentoring experiences of UGO and BAU ECTs were different across numerous roles, 
interactions, and impacts. 

e UGOECTs met with their mentors less frequently but for longer periods than BAU ECTs. 
They reported slightly higher levels of trust in their mentor than did BAU ECT's. When 
they did meet with their mentors, they received support from them in a variety of areas 
significantly more frequently than BAU ECTs received from theirs. 
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Intervention Findings 


The purpose of the intervention study was to examine the extent to which UGO mentors 
applied the mentoring model to their work with ECTs. In this study we distinguished 
implementation from intervention, with intervention defined as the interactions, activities and 
actions mentors engaged in with their ECTs. We examined intervention through a small 
exploratory study of UGO-mentored ECTs. The exploratory study drew on data from multiple 
sources to examine mentor-ECT dyad interactions among UGO ECTs who made strong gains in 
instructional practice, as measured by the Classroom Assessment Scoring System (CLASS), and 
those who did not. Using audio recordings of post-observation conversations, we used 
qualitative analyses techniques to examine differences in the mentoring intervention across 
mentor-ECT dyads. 


The analysis of mentor-UGO ECTs’ post-observation conversations revealed intriguing patterns 
in the intervention between ECTs who made the most gains on the CLASS (Gliders) and those 
who gained the least (Sliders). Mentor-ECT dyads with Glider ECTs engaged in significantly 
different ways from Slider dyads. ECTs in Glider dyads had longer conversations with their 
mentors, talked more about instruction and students, responded to each other more often, and 
engaged as peers more frequently than ECTs in Slider dyads. 


Glider dyads were also qualitatively different from Slider dyads in the actions they took. Glider 
dyads more frequently focused on solving problems of practice, targeting successes and 
strengths of the ECT, and mentors in Glider dyads more often redirected conversations to the 
positive, especially with regard to students. Glider dyad conversations picked up on challenges 
and addressed them, mentors directly facilitated conversations when needed. 


Impact Findings 


The purpose of this research was to estimate the impact of ECT participation in UGO. 
Specifically, we estimated the impact of the ECTs’ participation in UGO on teacher retention in 
the teaching profession in Alaska, instructional practice, and the academic achievement of 
ECTs’ students in reading, writing, and mathematics. While no statistically significant 
differences were found on the confirmatory contrasts (at the level of p = 0.05), the following 
findings emerged, with some promising effects: 

e Retention of UGO ECTs in their third year of teaching was higher than BAU ECTs (80.5% 
compared to 76.6%). While this finding indicates the treatment group had a higher 
retention rate than the control group, it represents a non-statistically significant 
difference with an effect size of 0.16. 

e Average gains on CLASS domains of emotional support, classroom organization, and 
instructional support were higher for BAU ECTs compared to UGO ECTs. This is the 
reverse of what we would hypothesize. Differences were not statistically significant, 
effect sizes ranged from -0.32 to -0.14. 
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e State assessment scores were higher for the primary students of Year 1 UGO ECTs in 
reading and mathematics compared to BAU ECTs. In writing, scores were higher for the 
primary students of Year 1 BAU ECTs than UGO ECTs. In Year 2 primary students of 
UGO ECTs attained higher scores in writing and mathematics. Students of BAU ECTs 
scored better in Year 2 in reading. No differences were statistically significant. 


e State assessment scores were higher for the secondary students of UGO ECTs in 
mathematics compared to students of BAU ECTs. While the differences were not 
statistically significant, after their first year of teaching, the effect size of 0.25 suggests 
substantively important differences. 


Exploratory analyses examined the effect of mentoring on the students of UGO ECTs by 
race/ethnicity. The secondary students of first year UGO ECTs who were identified as white 
(Caucasian) students, Hispanic students, Alaska Native students or students of two or more 
races (not Hispanic) obtained significantly higher scores on the state mathematics assessment 
than students with BAU teachers ( p values ranged from 0.008 to 0.037). American Indian 
students with first-year UGO-mentored ECTs also achieved higher scores on the state 
mathematics assessment than American Indian students with BAU teachers (p = 0.052). 


An additional exploratory analyses compared the students of ECTs who had UGO mentors 
with those in districts with no formal mentoring programs (three of the five districts). The 
primary students of first-year UGO ECTs obtained statistically significantly higher scores on the 
state reading assessment than students of BAU teachers (p = 0.019). 
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Chapter 7. Considerations/Implications 


Few studies of teacher mentoring have used research designs that employ treatment and control 
groups (Allen, 2005; Kraft et al., 2017). In our impact study we adhered to standards for 
rigorous educational research, using guidance from NE i3 and WWC to conduct a study using a 
randomized controlled trial. To provide context for interpreting the statistical results, we 
included studies not usually accompanying a rigorous study of impacts: a comprehensive study 
of implementation and a close look at the interactions between program mentors and ECT’s 
through the intervention study. Together these three studies enabled in-depth analyses of the 
UGO mentoring program. 


Overall, UGO was implemented with fidelity, although there is room to strengthen the 
intervention by focusing on educative mentoring in which mentor-ECT dyads collaboratively 
build on successes and address instructional practice. UGO teachers were retained in teaching 
at higher rates than BAU, although notat statistically significant levels. Analyses of student 
achievement suggest students of UGO-mentored ECTs generally had higher achievement 
scores, and the program had statistically significant and educationally important impacts on 
diverse secondary students in mathematics and primary grade students in reading. However, 
we found no statistically significant effects of UGO mentoring on instructional practice. The 
following sections are offered as considerations for UAF, as well as other mentoring programs, 
based on our research findings. 


Considerations for Mentoring Programs 


Some areas of the UGO program merit consideration as UAF continues to evolve and improve 
the ASMP model. One area is related to mentoring program expectations. In the ASMP rural 
model mentors visit ECTs monthly for at least 3.5 hours, which converted into an equivalent 
amount of time for face-to-face interaction under UGO. We found that some mentors struggled 
to consistently meet with all ECTs for this amount of time each month. Dedicated time with 
mentors and ECTs is essential for mentors to establish relationships, gather data on ECTs’ 
instruction, discuss results, and collaboratively identify next steps. Program leaders in urban 
settings should consider adapting the program to fit the busy schedules of urban teachers and 
to protect time for mentoring. 


UGO mentors were inconsistent in their use formative assessment tools when working with 
ECTs and in their documentation of how they used those tools. Formative assessment tools 
provide data for mentors to use in their discussions with mentees and as an entry point for 
mentors and ECTs to collaboratively agree upon potential strategies to solve instructional 
problems. Discussing data gathered during observations and collaboratively developing next 
steps is contingent upon dedicated interaction time. These are vital strategies that UAF and 
other mentoring programs can promote to strengthen effective educative mentoring. As 
program leaders and coaches review mentors’ CALs, they might track the extent to which 
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formative assessment tools are being used and keep tool-use in the forefront of mentors’ minds 
to increase the consistency of their use. 


Considerations for Mentoring Interactions with ECTs 


Our study detected little evidence that UGO impacted teachers’ instructional practices on 
domains of the CLASS. While ECT’s in UGO districts reported that their mentors strongly 
influenced their instructional practice, BAU ECT were rated stronger on classroom instruction 
as measured by the CLASS. These unexpected results may be due to inconsistency in the UGO 
intervention across mentors. Limited evidence from the intervention study suggests that some 
mentors may have engaged in social support of their teachers at the expense of educative 
mentoring. It is possible that focusing more closely on instructional practice—and particularly 
on student success — could produce more positive outcomes and effects that could be measured 
on instruments such as the CLASS. 


In light of our statistically nonsignificant estimates of the effects of mentoring on instructional 
practice, we looked to other lines of evidence for how mentors might shape classroom 
instruction. We found that ECTs whose instructional practice improved over the course of the 
study were in dyads in which mentors maintained an intentional and consistent focus on 
educative mentoring. Specifically mentors in these dyads: 


e Problem solved with new ECTs 

e Targeted successes and strengths of ECTs and their students 

e Listened for ECTs’ challenges and addressed them 

e Redirected/facilitated conversations as needed 

e Made time for mentor-ECT conversations 

e Focused on instruction and students 

e Ensured balanced air time in the conversation between mentors and ECTs 


e Allowed for differing opinions 


These practices echo other research on effective educative mentoring (Feiman-Nemser, 2001; 
Kraft et al., 2017; Lofthouse et al., 2010) and are directly applicable to mentors’ practice. Results 
of the intervention study suggest that ECTs’ classroom practice improves when mentors are 
more intentional in their coaching, following a cycle of planning for an observation, observing 
instruction and gathering data, interpreting results in collaboration with ECTs, co-developing 
next steps, and communicating between visits. 


Implications for Mentoring Programs to Bolster Impacts 


While we found that UGO was implemented with adequate fidelity, it might be that ideal 
fidelity is necessary to see the types of impacts the program suggests and mentoring programs 
seek. Coaching mentors is an area that emerged with implications for strengthening the entire 
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program and for ensuring that implementation is tightly focused on the model and on 
educative mentoring. Mentor communication with coaches was scored at an adequate level in 
Year 1 and Year 3 for most mentors, while most were at the ideal level in Year 2. Coaching 
challenges were reiterated during interviews. Coaches can play a critical role in supporting the 
mentors strive to improve their own practice and authentically engage with ECTs. However, 
more emphasis on coaches as instructional partners would require a shift in their current role, 
toward a more instructive relationship. It would also mean increased support for coaches from 
program leaders. One consideration is that during the UGO study, there were several changes 
in program leadership that may have influenced the work mentors did with ECTs and the role 
of coaches. A program such as UGO needs strong leadership to conceptually anchor the 
intervention and inspire mentors. 


Teacher retention was an area in which UGO ECTs experienced higher rates than BAU ECTs. 
Under the UGO model, the important connections between mentors and site administrators 
played a lesser role than in the ASMP model. Other research suggests that multiple supports are 
important for ECTs, and it is possible that the added support of principals through working 
relationships with mentors could boost retention further. Finding ways for mentors to engage 
meaningfully with site administrators, and possibly other teachers, might be a consideration to 
better situate mentoring in a teacher induction model and as a whole-school effort to improve 
teacher retention rates. Site administrators (principals) can support these efforts by protecting 
ECTs’ time for collaborating with mentors. 


The retention of several more teachers in Alaska each year likely has a positive effect 
economically, as it saves districts from hiring and training new teachers. It also provides 
stability to the teaching force in the state, facilitating long-term initiatives and professional 
learning. By using mentoring as part of a comprehensive induction package for new teachers, 
with support from principals, peers, and protected time for planning/mentoring, the impact 
could move beyond the effect size of 0.16 and show statistical significance. One consideration is 
that this study took place during a severe economic crisis in Alaska, which will likely continue 
into the coming school year. Current estimates are that for the 2017-18 school year the state will 
lose “about 123 ‘full-time equivalent’ positions, including 99 teachers, to save about $7.2 million 
total” (Hanlon, 2017, para. 3). In this economic climate, districts were forced to reduce teaching 
positions and cut back on other benefits, which may have influenced retention rates among both 
the UGO and BAU groups. 


Students of UGO ECTS attained higher scores on state assessments than students of BAU ECTs 
in several areas. As noted earlier, the trend toward higher scores for some students of UGO- 
mentored ECTs suggests that with a stronger intervention, results could be more pronounced. 
Our confirmatory and exploratory analyses, suggest some important findings related to student 
achievement. The differences in students’ secondary mathematics scores, after UGO ECTs’ first 
year of teaching, with the effect size of 0.25, suggests educational importance if not statistical 
significance. This, coupled with the statistically significant differences in outcomes for diverse 
secondary students in mathematics indicates UGO-mentored secondary mathematics ECTs 
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learned effective strategies for teaching mathematics to a diverse set of students. This is an 
important finding because algebra and secondary mathematics are essential for college success, 
particularly in the sciences, and improving mathematics outcomes for racial and ethnic minority 
students could increase their chances of entering and succeeding in postsecondary institutions. 
These results were attained without intentional matching of mentors to ECTs by teaching 
experience. UAF and other programs might consider the potential impact of assigning mentors 
to ECTs by content experience. 


Statistically significant effects for primary grade students of UGO ECTs in reading —as 
compared to BAU ECTs in districts that did not have formal, district-supported mentoring 
programs—suggest the impact UGO mentoring can have. UGO mentoring can help boost the 
reading achievement of primary grade students, which is crucial for their academic success. 
Ensuring that every child is a reader has been the goal of instruction, educational research, and 
reform efforts for decades. This is an important finding with implications for the value of 
supporting new teachers through mentoring. It shows that mentors have a significant lever for 
improving ECT practice and ultimately impacting student achievement in reading. 


A final consideration is that both statistically significant exploratory findings came after ECTs’ 
first year of teaching. This prompts questions about both how to further strengthen mentoring 
in the first year and how to deepen it in the second year. How could effects seen among 
students of first-year teachers be extended to more students? Is it possible that second-year 
mentor training could be modified to continue to “push” teachers’ practice as they gain 
experience? Might such training produce a measureable impact? These questions warrant 
discussion of mentor training in ECTs’ second year and how best to support the mentors of 
ECTs as they grow in their practice. 


78 Education Northwest 


References 


Achinstein, B. & Athanases, S.Z. (2005). Focusing new teachers on diversity and equity:Toward 
a knowledge base for mentors. Teaching and Teacher Education, 21, 843-862. 


Alaska Department of Education & Early Development. (n.d.). Alaska Standards. Retrieved from 
https://education.alaska.gov/standards 


Alaska Department of Education & Early Development, Alaska Comprehensive Center, Alaska 
Native Educators, & Education Northwest. (2012.). Guide to implementing the Alaska 
Cultural Standards for Educators. Juneau, AK: Authors. Retrieved from 
https://www.kegbsd.org/cms/lib/AK01001769/Centrici oduleInstance/9459/Alaska%2 


OCultural%20Standards.pdf 


Allen, J. P., Pianta, R. C., Gregory, A., Mikami, A. Y., & Lun, J. (2011). An interaction-based 
approach to enhancing secondary school instruction and student achievement. Science, 
333(6045), 1034-1037. 


Allen, M. B. (2005). Eight questions on teacher recruitment and retention: What does the research say? 
Retrieved from ERIC database. (ED489332) 


Borman, G. D., & Dowling, N. M. (2008). Teacher attrition and retention: A meta-analytic and 
narrative review of the research. Review of Educational Research, 78(3), 367-409. 


Boyer, R. D. (2012). Certificated Accountability Report: Report to the Anchorage School District 
Superintendent on ASD. Anchorage, AK: Anchorage School District. 


Century, J., & Cassata, A. (2016). Implementation research: Finding common ground on what, 
how, why, where, and who. Review of Research in Education, 40(1), 169-215. 


The Danielson Group & Teachscape. (2013). Introduction to the framework for teaching: A training 
program for teachers, facilitation guide. Retrieved August 4, 2017 from 
https://focus.teachscape.com/pdf/FFT_TrainingProgram_FacilitationGuide.pdf 


DeCesare, D., McClelland, A., & Randel, B. (2017). Impacts of the Retired Mentors for New Teachers 
program (REL 2017-225). Retrieved from U.S. Department of Education, National Center 
for Education Statistics website: 
https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2017225 


DeFeo, D. J., Tran, T., Hirshberg, D., Cope, D., & Cravez, P. (2017). The cost of teacher turnover in 
Alaska. Retrieved from University of Alaska Anchorage, Institute of Social and Economic 


Research website: http://www. iser.uaa.alaska.edu/Publications/2017-CostTeacher.pdf 


Mentoring ECTs in Urban Alaska 79 


Dunst, C. J., Trivette, C. M., & Raab, M. (2013). An implementation science framework for 
conceptualizing and operationalizing fidelity in early childhood intervention studies. 
Journal of Early Intervention, 35(2), 85-101. 


Feiman-Nemser, S. (2001). Helping novices learn to teach: Lessons from an exemplary support 
teacher. Journal of Teacher Education, 52(1), 17-30. 


Fixsen, D. L, Naoom, S. F,, Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation 
research: A synthesis of the literature. Retrieved from University of North Carolina, Chapel 
Hill, FPG Child Development Institute, National Implementation Research Network 
website: http://nim.fpg.unc.edu/resources/implementation-research-synthesis-literature 


Goldrick, L. (2009). A teacher development continuum: The role of policy in creating a supportive 
pathway into the profession [Policy brief]. Santa Cruz, CA: New Teacher Center. 


Greene, J. A. (2015). Serious challenges require serious scholarship: Integrating implementation 
science into the scholarly discourse. Contemporary Educational Psychology, 40, 112-120. 


Guarino, C. M., Santibanez, L., Daley, G. A. (2006). Teacher recruitment and retention: A review 
of the recent empirical literature. Review of Educational Research, 76(2), 173-208. 


Hanlon, T. (2017, February 22). Anchorage School Board cuts dozens of teaching positions, and 
the situation could get even worse (Updated ed.). Alaska Dispatch News. Retrieved from 
https://www.adn.com/alaska-news/education/2017/02/21/anchorage-school-board- 


passes-a-budget-that-cuts-dozens-of-teaching-positions/ 


Harn, B. A., Damico, D. P., & Stoolmiller, M. (2017). Examining the variation of fidelity across 
an intervention: Implications for measuring and evaluating student learning. Preventing 
School Failure. Advance online publication. 


Hirshberg, D., & Hill, A. (2013, April). Teacher supply and demand update 2013. PowerPoint 
presentation to the Alaska Teacher Placement Advisory Board, Anchorage, AK. 


Hobson, A. J., Ashby, P., Malderez, A., & Tomlinson, P. D. (2009). Mentoring beginning 
teachers: What we know and what we don't. Teaching and Teacher Education, 25(1), 207— 
216. 


Hoy, W. K., & Tschannen-Moran, M. (1999). The five faces of trust: An empirical confirmation in 
urban elementary schools. Journal of School Leadership, 9(3), 184-208. 


Ingersoll, R. M. (2001). Teacher turnover and teacher shortages: An organizational analysis. 
American Educational Research Journal, 38(3), 499-534. 


80 Education Northwest 


Ingersoll, R., & Kralik, J. M. (2004). The impact of mentoring on teacher retention: What the research 
says [Research review]. Retrieved from University of Pennsylvania, Graduate School of 


Education website: https://www.gse.upenn.edu/pdf/rmi/ECS-RMI-2004. pdf 


Ingersoll, R., Merrill, L., & Stuckey, D. (2014). Seven trends: The transformation of the teaching force. 
Consortium for Policy Research in Education. http://eric.ed.gov/?id=ED566879 


Ingersoll, R. M., & Strong, M. (2011). The impact of induction and mentoring programs for 
beginning teachers: A critical review of the research. Review of Education Research, 81(2), 
201-233. 


Kekahio, W., Cicchinelli, L., Lawton, B., & Brandon, P. R. (2014). Logic models: A tool for effective 
program planning, collaboration, and monitoring (REL 2014-025). Retrieved from U.S. 
Department of Education, National Center for Education Statistics website: 
https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2014025 


Knight, J. (2009). Instructional coaching: A multimedia kit for professional development. Thousand 
Oaks, CA: Corwin. 


Knowles, M. S., & Associates. (1984). Andragogy in action: Applying modern principles to adult 
learning. San Francisco, CA: Jossey-Bass. 


Kraft, M. A., Blazar, D., & Hogan, D. (2017). The effect of teaching coaching on instruction and 
achievement: A meta-analysis of the causal evidence [Working paper]. Retrieved from 
Harvard University, Scholars at Harvard website: 


achievement-meta-analysis-causal 


Lofthouse, R., Leat, D., Towler, C., Hall, E., & Cummings, C. (2010). Improving coaching: 
Evolution not revolution [Research report]. Reading, Berkshire, UK: CfBT Education Trust. 


Merriam, S. B. (2001). Andragogy and self-directed learning: Pillars of adult learning theory. 
New Directions for Adult & Continuing Education, 89, 3-14. 


Mitchell, R. M., Celano, F., & Tarter, C.J. (2009, November). Teacher trust of mentor scale 
development. Paper presented at the annual convention of the University Council for 
Educational Administration, Anaheim, CA. 


Murkowski, L. (2003, April 25). Speech to Alaskan Legislature. Retrieved from Iowa State 
University, Archives of Women's Political Communication website: 
https://awpc.cattcenter.iastate.edu/2017/03/21/speech-to-alaskan-legislature-april-25- 


Mentoring ECTs in Urban Alaska 81 


National Commission on Teaching & America’s Future. (2016). What matters now: A new compact 
for teaching and learning — the evidence base. National Commission on Teaching and 


America’s Future https://eric.ed.gov/?id=ED572506 


Pianta, R., & Hamre, B. (2008). CLASS®: The Classroom Assessment Scoring System®. 
Charlottesville, VA: University of Virginia, Curry School of Education, Center for 
Advanced Study of Teaching and Learning. 


Ross-Gordon, J. M. (2011). Research on adult learners: Supporting the needs of a student 
population that is no longer nontraditional. Peer Review, 13(1). Retrieved from 
https://www.aacu.org/publications-research/periodicals/research-adult-learners- 


supporting-needs-student-population-no 


Sawchuk, S., & Rebora, A. (2016, October 4). New teachers make up a significant segment of 
profession. Education Week, 36(7), pp. 1, 14. Retrieved from 
http://www.edweek.org/ew/articles/2016/10/05/new-teachers-make-up-a-significant- 


segment.html 


Smith, M. K. (2002). Malcolm Knowles, information adult education, self-direction and andragogy. 
Retrieved from YMCA George Williams College, infed website: 
http://infed.org/mobi/malcolm-knowles-informal-adult-education-self-direction-and- 


andragogy/ 


Sprick, R., Knight, J., Reinke, W., Skyles, T. M., & Barnes, L. (2010). Coaching classroom 
management: Strategies and tools for administrators and coaches (2nd ed.). Eugene, OR: 
Pacific Northwest. 


Strauss, S. E., Johnson, M. O., Marquez, C., & Feldman, M. D. (2013). Characteristics of 
successful and failed mentoring relationships: A qualitative study across two academic 
health centers. Academic Medicine, 88(1), 82-89. Retrieved from 
https://www.ncbi.nlm.nih.gov/pmce/articles/PMC3665769, 


Strong, M. (2009). Effective teacher induction and mentoring: Assessing the evidence. New York, NY: 
Teachers College Press. 


Tschannen-Moran, M. (2017). The importance of trust in schools. WERA Educational Journal, 9(2), 
3-11. 


U.S. Department of Education, Institute of Education Sciences, National Center for Education 
Evaluation and Regional Assistance, What Works Clearinghouse. (2014). What Works 
Clearinghouse™: Procedures and standards handbook, version 3.0. Retrieved from 
https://ies.ed. gov/ncee/wwc/Docs/referenceresources/wwc_ procedures v3 0 standards 


handbook.pdf 


82 Education Northwest 


U.S. Department of Education, Institute of Education Sciences, National Center for Education 
Evaluation and Regional Assistance, What Works Clearinghouse. (2016). Review protocol 
for teacher training, evaluation, and compensation, version 3.2. Retrieved from 
https://ies.ed. gov/ncee/wwc/Docs/ReferenceResources/wwc_ttec_ protocol v3.2.pdf 


U.S. Department of Education, National Center for Education Statistics. (2012). Common Core 
of Data. Local Education Agency (School District) Universe Survey Data, 2010-11 (v.2a). 


Retrieved October 9, 2012, from https://nces.ed.gov/ccd/pubagency.asp 


W. K. Kellogg Foundation. (2004). Logic model development guide: Using logic models to bring 
together planning, evaluation, and action: (Updated ed.). Battle Creek, MI: Author. 


Mentoring ECTs in Urban Alaska 83 


Appendix A. ECT Survey Data Tables—Combined 
Responses from Year 1 and Year 2 Treatment (UGO) 


and Control (BAU) ECTs 


Throughout this document, an asterisk ( 


(p< 0.05). 


) indicates a statistically significant difference 


Table A-1. Training, professional development, and other support ECTs received 


What types of training, professional development, and/or support 
have you received this year (select all that apply)? 


New teacher orientationt 
District/school culture orientationt 
District/school curriculum training* 


District/school professional development 

Release time to participate in professional development 
Release time to observe other teachers 

Support form district/school content coaches 

Support from liaison from teaching program 

Common grade-level planning time 

Common subject-level planning time 

Collaboration time with other new teachers 


Percentage responding “Yes” (n) 


BAU UGO 
44.3% (155) | 43.6% (159) 


21.7% (76) 17.3% (63) 

61.9% (273) | 57.3% (262) 
87.5% (448) | 74.5% (400)* 
50.2% (257) | 41.7% (224)* 
42.6% (218) | 25.1% (135)* 
39.5% (202) | 29.1% (156)* 


) 
41.3% 222)* 
24.8% (133) 
25.0% (134) 


) 
55.1% 282) 
27.2% (139) 
29.5% (151) 


( ( 
( (1 
( (1 
(22 (29 
( ( 
( ( 


tCohort1 treatment ECTs were notasked this in year 1; results exclude cohort 1, year 1 treatment and control ECTs. 


Table A-2. Sources of support the ECT received most to improve effectiveness in the classroom 


From what source do you receive the most support to improve your 


effectiveness in the classroom? 


A mentor* 


District professional development* 
Mandated collaboration with colleagues* 
Informal collaboration with colleagues* 


Site administrator or principal* 
Former host teacher* 
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Percentage responding “Yes” (n) 


BAU 
25.9% 


( 
4.9% ( 
4.9% ( 
45.7% (233) 
7.7% ( 
2.9% ( 


Table A-3. Number of times a site administrator/principal visited an ECT classroom for at least 5 
to 10 minutes to observe instructional activities 


How many times has your site administrator/principal been in your Mean (S.D). 
classroom for at least 5-10 minutes to observe instructional n 
ere : > 
activities this year: BAU UGO 
5.6 (5.28) 6.6 (5.82)* 
480 504 


Table A-4. Frequency of time spent with other ECTs in the ECT’s school and district 
How much time do you spend with Percentage (n) 
other early career teachers in your ; 
school? In your district? None Very little Some Alot 
School 


BAU 15.1% (72) 33.3% (159) 33.3% (159) 18.4% (88) 

UGO 23.1% (115) 32.6% (162) 29.2% (145) 15.1% (75)* 
District 

BAU 33.9% (171) 43.9% (221) 19.4% (98) 2.8% (14) 

UGO 33.5% (179) 46.5% (249) 15.7% (84) 4.3% (23) 


Table A-5. ECTs’ agreement on instructional context 


To what extent do you agree with Strongly : Strongly 
the following statements? disagree Dlsegiee AgICS agree 
My colleagues contribute to my BAU (0.6% (3) 3.3% (17) | 47.1% (241) | 49.0% (251) 
professional growth. UGO 0.9% (5) 2.6% (14) | 51.6% (277) | 44.9% (241) 
My site administrator/principal BAU —.2.0% (10 2.7% (14) | 51.5% (263) | 43.8% (224) 
supports my professional growth. UGO 2.2% (10 5.2% (28) | 53.5% (287) | 39.1% (210) 
| accept and feel comfortable with BAU 0.8% (4) 3.3% (17) | 47.8% (244) | 48.1% (246) 
the contare ofthe communtyin — UGO. 0.9% () | 3.7% (20) | 51.6% (277) | 43.8% (235) 
| accept and feel comfortable with BAU 0.8% (4) 7.1% (36) | 47.6% (242) | 44.5% (226) 
in which | 

the climate of the schoolinwhich! cq 4.3% (7) | 7.3% (39) | 51.3% (274) | 40.1% (214) 
Connecting my teaching to the BAU 0.7% (3) 2.7% 12) | 44.9% (198) | 51.7% (228) 
student's cultural background is gq g. 4% (2) | 1.5% (7) | 46.0% (210) | 52.1% (238) 


essential to student learning. 
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Table A-6. Supports that enhanced ECTs’ ability to teach students 
Which of the following supports have enhanced Percentage (n) 
your ability to teach students this year (select all 


that apply)? Please rank the top three supports. Percentage responding “Yes” (n) 


Ranked in the top three 


Support BAU UGO BAU ranked UGO ranked 
Formal mentor 49.0% (251) 94.0% (505)* 77.3% (194) 74.7% (377) 
Informal mentor 41.2% (211) 33.3% (179)* 66.4% (140) 58.1% (104) 
District-level support 19.5% (100) 17.3% (93) 27.0% (27) 14.0% (13)* 
Common planning time with other teachers 50.0% (256) 43.4% (233)* 53.1% (136) 35.6% (83)* 
Grade-level collaborations 49.4% (253) 48.6% (261) 57.7% (146) 44.8% (117)* 
Administrator support 55.9% (286) 48.2% (259)* 54.6% (156) 38.6% (100)* 
Aide(s) paraprofessional(s) 34.4% (176) 34.3% (184) 56.8% (100) 46.7% (86) 
Support from students’ parents 36.7% (188) 33.2% (178) 22.9% (43) 18.0% (32) 
Sufficient salary 30.3% (155) 26.6% (143) 18.1% (28) 15.4% (22) 
Sufficient resources and materials 44.9% (230 45.6% (245) 40.0% (92) 24.9% (61)* 
Informal collaborations 57.6% (295) 60.2% (323) 53.9% (159) 34.4% (111)* 
University programs 13.1% (67) 15.8% (85) 35.8% (24) 18.8% (16)* 
Professional development/workshops/ 57.2% (298) 60.9% (327) 48.8% (143) 39.5% (129)* 
conferences/training 

Extra resources (volunteers ) 23.1% (118) 19.2% (103) 22.9% (27) 8.7% (9)* 


Note: Question varied per survey administration; in 2015 and 2016, question reads: “Describe any mentor support you had this year (selectall that apply)?” 
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Table A-7. Challenges that hindered ECTs’ ability to teach students 


Which of the following challenges have Percentage (n) 

hindered your ability to teach students this year 

pee all that apply)? Please rank the top Percentage responding “Yes” (n) Ranked in the top three 
Challenge BAU UGO BAU UGO 
Time management 29.1% (149) 29.6% (159) 47.7% (71) 41.5% (66) 
Time constraints or school schedules 41.4% (212) 45.3% (243) 67.0% (142) 58.4% (142) 
Classroom management 29.5% (151) 30.0% (161) 67.6% (102) 48.5% (78)* 
Lack of administrative support 9.2% (47) 14.2% (76)* 63.8% (30) 59.2% (45) 
Inexperience with culture 3.9% (20) 4.8% (26) 35.0% (7) 19.2% (5) 
Conflict in personality with mentor 0.6% (3) 0.6% (3) 66.7% (2) 33.3% (1) 
Being assigned a mentor with no choice 0.8% (4) 1.5% (8) 75.0% (3) 37.5% (3) 
Not having a mentor 10.4% (53) 0.4% (1)* 0.0% (0) 0.0% (0) 
Relationship with parents 8.0% (41) 12.3% (66)* 43.9% (18) 39.4% (26) 
Isolation 12.3% (63) 8.9% (48) 49.2% (31) 43.8% (21) 
Low student motivation 39.3% (201) 39.7% (213) 77.6% (156) 61.5% (131)* 
Student attendance 40.8% (209) 39.7% (213) 65.1% (136) 51.6% (110)* 
Student personal issues 38.5% (197) 35.8% (192) 58.4% (115) 54.2% (104) 
Stress 40.0% (205) 38.2% (205) 60.5% (124) 47.3% (97)* 
Phases/level system 1.8% (9) 0.7% (4) 44.4% (4) 25.0% (1) 
Curriculum 16.0% (82) 19.7% (106) 53.7% (44) 43.4% (46) 
Inadequate teacher preparation 5.7% (29) 8.0% (43) 34.5% (10) 39.5% (17) 
Wide range of grades/levels 19.5% (100) 23.9% (128) 56.0% (56) 51.6% (66) 
Insufficient salary 6.8% (35) 6.7% (36) 42.9% (15) 50.0% (18) 
Differentiating instruction for diverse learners 34.8% (178) 31.1% (167) 53.4% (95) 44.9% (75) 
Grade-level collaborations 2.7% (14) 5.6% (30)* 50.0% (7) 26.7% (8) 
Lack of resources (people or objects) 23.1% (118) 20.9% (112) 58.5% (69) 54.5% (61) 
School culture 9.0% (46) 10.8% (58) 39.1% (18) 46.6% (27) 
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Table A-8. ECTs who reported having a mentor (formal or informal) 


Did you have a formal/informal mentor? Percentage reporting “Yes” (n) 
BAU 70.1% (359) 
UGO 98.7% (530)* 


Table A-9. Start dates for mentoring 


When did you begin working with your Percentage (n) 

mentor? August/September After September 
BAU 80.5% (285) 19.5% (69) 
UGO 95.8% (429) 4.2% (19)* 


Table A-10. ECT success attributed to mentor 


Percentage (n) 
Of the success you've had as an early career 


teacher, what proportion would you attribute Not at all/ San Quite a bit/a 
to help from your mentor? hardly at all . great deal 
BAU 8.4% (30) 37.3% (134) 54.3% (195) 
UGO 4.2% (22) 30.1% (159) 65.8% (348)* 


Table A-11. Perceived mentor responsibility for formal evaluation of ECTs 
Does your mentor have any responsibility for 


formally evaluating you (e.g., feedback to Percentage (n) 
your principal)? 

| don’t know No Yes 
BAU 18.1% (65) 79.1% (284) 2.8% (10) 
UGO 17.8% (80) 76.8% (345) 5.4% (24) 


Table A-12. ECT-mentor supports provided by school/district 
What supports has your district/school provided you to work with —Percentage responding “Yes” (n 


your mentor (select all that apply)? BAU UGO 

| have access to a substitute. 46.5% 9.8% (52)* 
My mentor has access to a substitute. 34.0% (122) 0.4% (1)* 
We share acommon planning time. 9.5% (34 23.1% (59)* 
We share acommon teaching assignment (grade or subject). 29.5% 4.3% (11)* 
We are within close proximity to each other. 31.8% 5.1% (13)* 
Pa released from non-instructional duties (bus, playground, 0.6% (2 7.4% (39)* 
My mentor is released from non-instructional duties. 2.0% 3.5% (9)* 
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Table A-13. Similarity of ECTs’ and mentors’ teaching experience 


My mentor and | have the same experience teaching in the _Percentage responding “Yes” (n) _ 
following situations (select all that apply). BAU UGO 

Same grade 62.1% (223 32.3% (171)* 
Same school level 41.8% (150 26.2% (139)* 
Same subject(s) 51.5% (185 35.3% (187)* 
Similar student populations, such as SPED or ELL 47.9% (172 34.0% (180)* 
Same district 81.3% (292 77.4% (410) 
Same school 33.7% (121 8.1% (43)* 


Table A-14. Issues addressed by mentors 

My mentor addresses the following issues with me (select all that __Percentage responding "Yes" (n) _ 

UGO 

76.0% (403 
93.8% (497 
69.4% (368) 
77.7% (412) 
47.7% (253) 
64.2% (340)* 


* 


=p ag ogy 

Classroom management 

Content matter 

Curriculum materials 
School-specific logistics/pro cedures 
District-specific logistics/procedures 


* 


~~ a 


Table A-15. ECTs’ characterizations of their mentors 


| would characterize my mentor as a/an....(select all that apply) err exesporidin rs n 


Colleague 82.2% ( 68.7% (364) 
Role model 66.0% (237) 72.1% (382)* 
Evaluator 12.5% (45 31.5% (167)* 
Therapist/counselor 19.8% ( 34.5% (183)* 
Friend 58.5% ( 62.1% (329) 
Expert guide 55.2% ( 72.8% (386)* 
Advocate 49.9% ( 65.5% (347)* 
Critic 8.6% (31) 13.4% (71)* 


Table A-16. ECTs’ preferred methods of communication with mentor 


My preferred method(s) of communication with my mentoris ... Percentage responding “Yes” (n) 
(select all that apply) 


BAU UGO 
Face-to-face 91.1% (327) 87.9% (466) 
Telephone/skype audio 16.2% (58 10.6% (56)* 
Email 58.5% ( 72.1% (382)* 
Chat/text 26.5% (95 20.6% (109)* 
Skype video/video conferencing 3.6% (13 1.5% (8)* 
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Table A-17. Predominate nature of ECTs’ communications with mentor 


Our [mentor and ECT] communications are Percentage (n) 

mostly ... Formal Informal Both 
BAU 27.7% (99) 25.4% (91) 46.9% (168) 
UGO 42.8% (227) 3.2% (17) 54.0% (286)* 


Table A-18. ECTs’ preferred communication Time 
Percentage responding “Yes” (n) 


My mentor and | usually meet together ... (select all that apply) 
BAU UGO 


Before school 19.8% (71 11.9% (63)* 
During class time 13.9% (50 54.0% (286)* 
During planning time 25.6% (92 79.3% (420)* 
During lunch 16.7% (60 52.6% (279)* 
After school 76.0% (27 54.9% (291)* 
On non-school days 18.4% (66 6.8% (36)* 


Table A-19. Convenience of ECTs’ regular meeting time with mentor 


The meeting times my Percentage (n) 

inentorenG cavers : soy : Inconvenient sometimes Convenient Nor. 

te inconvenient convenient convenient 
BAU 0.3% (1) 0.6% (2) 19.6% (70) 48.7% (174) 30.8% (110) 
UGO 0.8% (4) 0.2% (1) 16.2% (85) 43.8% (230) 39.1% (205) 
Table A-20. Frequency of ECTs’ face-to-face contact with mentor* 

My mentor and | Percentage (n) 

are in contact : E t ; 

face-loface ... Daily Weekly iat Monthly Occasionally Never 
BAU 15.9% (57) 15.6% (56) 34.3% (123) 25.1% (90) 8.9% (32) 0.3% (1) 
UGO 0.0% (0) 9.1% (48) 45.3% (240) 44.2% (234) 1.5% (8) 0.0% (0) 


Table A-21. Length of time for face-to-face meetings* 


On average, Percentage (n) 

these sessions 15 minutes About About About About More than 

last ... orless 30 minutes 1 hour 2 hours 3 hours 3 hours 

BAU 24.0% (86) 21.8% (78) 39.4% (141) 12.6% (45) 0.8% (3) 1.4% (5) 
0.6% (3) 19.1% (101) 34.0% (180) 24.3% (129) 14.2% (75) 7.9% 


Table A-22. Influence of face-to-face meetings on ECTs’ teaching practice 


This face-to-face contact Percentage (n) 


i i i Not at all/ Quite a bit/ 
influences my teaching practice ... hardly at all Some aires deal 
BAU 3.4% (12) 35.2% (123) 61.3% (214) 
UGO 2.5% (13) 25.0% (130) 72.6% (378)* 
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Table A-23. Frequency of contact with ECTs through distance methods* 


My mentor and | Percentage (n) 

are in contact 

does Daily Weekly on Monthly Occasionally Never 
methods* ... 

BAU 5.6% (19) 43.8% (148) 22.8% (77) 11.0% (37) 16.9% (57) 0.0% (0) 
UGO 0.8% (4) 74.3% (394) 19.4% (103) 3.8% (20) 1.7% (9) 0.0% (0)* 


*Such as email, phone, Skype, instant messaging. 


Table A-24. Influence of distance meetings on ECTs’ teaching practice 


The distance contact influences peteentage sn) 


; ; Not at all/ Quite a bit/ 
a hardly at all mone a great deal 
BAU 17.7% (53) 47.0% (141) 35.3% (106) 
UGO 10.0% (13) 47.4% (237) 42.6% (213)* 
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Table A-25. Frequency and extent of influence of mentor/mentee activities 


The following are activities your mentor could 


do when visiting with you. For each activity, 
indicate how often it occurs and to what 
extent it influences your teaching practice. 


Activities 
Ob instructi aoe 
serve your instruction 
UGO 
BAU 
Gather classroom data 
UGO 
Model lessons or strategies with BAU 
your students and/or co-teach in 
your classroom UGO 
Provide you with resources and BAU 
materials UGO 
D nt k togeth ale 
ocument your work to er 
” a“ UGO 
Engage with you in goal setting 
process (for example, use an BAU 
individual learning plan, UGO 
professional growth plan, 
professional development plan, etc. 
Brainstorm with you ways to BAU 
approach a challenge with a UGO 


student or class 
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Never/ 
occasionally 


89.1% (317) 
11.4% (60) 
84.2% (298) 
24.4% (129) 
92.7% (330) 
79.7% (408) 
44.3% (158) 
24.3% (129) 
43.1% (154) 
7.0% (37) 


56.9% (203) 
15.5% (82) 


30.1% (107) 
11.2% (59) 


Frequency 


Monthly 


20.7% (74) 
42.3% (224) 


18.5% (66) 
29.9% (158) 


Every two 
weeks/ 
weekly 


3.1% (11) 
36.4% (192)* 
5.4% (17) 
31.2% (165)* 
3.1% (11) 

8.2% (42)* 
36.1% (129) 
46.2% (145)* 
37.5% (134) 
59.1% (313)" 


22.4% (80) 
42.2% (223)* 


51.4% (183) 
59.0% (312)* 


Percentage (n) 


Influence on teaching practice 


Not at all/ 
hardly at all 


8.7% (14) 


Some 


44.7% (72) 
26.2% (135) 
50.0% (72) 
31.9% (159) 
33.3% (36) 
33.0% (93) 
33.6% (113) 
28.4% (145) 
41.1% (97) 
35.5% (179) 


39.6% (97) 
26.6% (138) 


25.0% (85) 
13.6% (71) 


Quite a bit/ 
a great deal 


46.6% (75) 
70.5% (363)* 
39.6% (57) 
65.7% (327)* 
61.1% (66) 
65.6% (185) 
62.8% (211) 
69.3% (354) 
39.4% (93) 
58.1% (293)* 


55.9% (137) 
70.3% (365)* 


72.7% (247) 
85.1% (446)* 


Table A-26. Frequency and extent of influence of mentor/mentee topic discussions 


The following are topics you might discuss 


with your mentor. For each topic, indicate how 


often these conversations occur and to what 
extent they influence your teaching practice. 


Topic 


Observation of your instruction and/or 
data that were gathered 


Issues of equity (e.g., in environment 
or atmosphere; how students are 
engaged; curriculum, content, 
assessments) 


Cultural awareness, values, and 
sensitivity 


Working with special populations 
(e.g., learning disabled, English 
language learners, gifted and 
talented, physically handicapped) 


Lesson planning 
Parent communication 


Site administrator/principal 
communication 


BAU 
UGO 


BAU 
UGO 


BAU 
UGO 


BAU 
UGO 


BAU 
UGO 
BAU 
UGO 


BAU 
UGO 


Never/ 
occasionally 


77.5% (276) 
11.1% (50) 


46.5% (166) 
18.9% (100) 


67.2% (240) 
44.2% (234) 


Frequency 


Monthly 


11.8% (42) 
46.3% (208) 


21.6% (77) 
39.1% (207) 


14.3% (51) 
31.0% (164) 


19.3% (69) 
28.9% (153) 


( 

( 
20.5% (73) 
28.2% (148) 
18.8% (67) 
28.3% (150) 
15.4% (55) 
25.7% (136) 


Percentage (n) 


Every two 
weeks/ 
weekl 

10.7% (38) 


42.5% (191)* 


39.1% (114) 
42.1% (223)* 


18.5% (66) 
24.8% (148)* 
29.1% (104) 
32.6% (173)* 
25.5% (91) 


Influence on teaching practice 


Not at all/ 
hardly at all 

6.1% (12) 

1.6% (7) 


3.8% (11) 
3.2% (16) 


Some 


48.5% (95) 
20.1% (88) 


43.9% (127) 
29.6% (149) 


53.6% (135) 
46.4% (221) 


43.8% 
36.9% 


126 
178 


44.3% 
40.0% 
52.0% 
46.7% 


49.4% 
45.6% 


129 
183 
143 
214 


125 


) 
) 
) 
) 
) 
) 
) 
197) 


( 
( 
( 
( 
( 
( 
( 
( 


Quite a bit/ 
a great deal 


45.4% (89) 
78.4% (344)* 


52.3% (151) 
67.3% (339)* 


38.5% (97) 
47.1% (224) 


51.7% (149 
58.7% (283 


(149) 

(283) 
48.1% (140) 
53.1% (243) 
43.3% (119) 
45.6% (209) 
40.3% (102) 
46.3% (77) 
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Table A-27. Mentors’ responsiveness to ECT needs 
Overall, how 
responsive has your 
mentor been to your 
needs this year? 

BAU 8.4% (30) 14.9% (53) 76.7% (273) 
UGO 10.9% (58) 6.4% (34) 82.6% (438)* 


Percentage (n) 


Very unresponsive Somewhat responsive Very responsive 


Table A-28. ECTs’ agreement on work with mentors 


To what extent do you agree Percentage agreement 


with the following statements? Strongly Disagree Agree Strongly 
disagree agree 

My work with my mentor is BAU 0.9% (3) 3.4% (12) | 38.7% (137) | 57.1% (202) 
guided by professional UGO 76.3% 
teaching standards. 0.0% (0) | 0.4% (2) | 23.3% (123) | (4goy 
My work with my mentor is BAU 3.1% (9 9.4% (27) | 47.0% (135) | 40.4% (116) 
dri tly by relationshi 
and not paperwork. | DCO 1.5% (8) | 10.1% (53) | 47.4% (249) | 41.0% (215) 

51.7% 
My mentor provides emotional oe 1.4% (5) 6.5% (23) | 40.4% (143) (183)" 
support: UGO 0.4% (2) 2.9% (15) | 37.3% (196) | 59.4% (312) 
My mentor supports my BAU 0.7% (2) 4.5% (14) | 46.5% (144) | 48.4% (150) 
understanding of the culture(s) UGO 56.3% 
in my community. 0.0% (0) 2.3% (10) | 41.4% (184) (250)* 
A formal definition of mentor- = BAU = 46.2% (32) | 46.7% (92) | 25.9% (51) | 11.2% (22) 
mentee roles would have been 
helpful. t UGO = 12.6% (45) | 43.6% (156) | 32.4% (116) | 11.5% (41) 
My work with my mentor BAU 
includes the content, 1.5% (5) 11.3% (39) | 46.4% (160) | 40.9% (141) 
performance, and/or cultural UGO 
standards for Alaska's 0.2% (1) | 3.3% (17) | 34.8% (180) | 1.8% 
students. (320) 

10) 10) 10) 10) 

Overall, havings mentor has fees 0.6% (2) 2.8% (10) | 30.1% (106) eee (234) 
been beneficial to my teaching. 0.6% (3) 3.0% (16) | 17.8% (94) matey 


t Question was notasked in 2016 survey administration. 
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(415)* 


Table A-29. ECTs’ agreement on trust scale 


The following are statements about your mentor. Please 


indicate the extent to which you agree or disagree with each 


statement... 


Honesty 

| trust my mentor. 

| have faith in the integrity of my mentor. 

My mentor keeps his or her word. 

When my mentor tells me something | can believe it. 
Benevolence 


My mentor typically looks out for me. 


My mentor typically acts with my best interest in mind. 


My mentor shows concern for me. 

My mentor is unresponsive to my concerns.! 
Competence 

| am suspicious of most of my mentor's actions.* 

My mentor is competent in doing his or her job. 


Reliability 


Even in difficult situations | can depend on my mentor. 


My mentor is reliable. 
Openness 


My mentor is open. 


My mentor openly shares personal information with me. 


TOTAL SCORE 


Mean (S.D) 

BAU UGO 
5.7 (0.59) 5.8 (0.47)* 
(356) (530) 
5.7 (0.72) 5.8 (0.51)* 
(356) (530) 
5.7 (0.61) 5.8 (0.67) 
(356) (530) 
5.7 (0.59) 5.8 (0.50)* 
(356) (530) 
5.6 (0.78) 5.7 (0.70)* 
(356) (530) 
5.6 (0.64) 5.8 (0.53)* 
(356) (530) 
5.6 (0.85) 5.8 (0.53)* 
(356) (530) 
5.7 (0.68) 5.8 (0.51)* 
(356) (530) 
5.6 (0.77) 5.8 (0.57)* 
(356) (530) 
5.6 (0.96) 5.7 (0.97) 
(356) (530) 
5.7 (0.64) 5.8 (0.57)* 
(356) (530) 
5.8 (0.81) 5.8 (0.84) 
(356) (530) 
5.7 (0.74) 5.8 (0.55)* 
(356) (530) 
5.6 (0.77) 5.8 (0.56)* 
(356) (530) 
5.5 (0.91) 5.7 (0.67)* 
(356) (530) 
5.6 (0.73) 5.8 (0.55)* 
356 530 
5.2 (0.90) 5.2 (0.76) 
(356) (530) 
5.7 (0.70) 5.8 (0.51)* 
(356) (530) 
4.8 (1.44) 4.6 (1.36) 
(356) (530) 
5.6 (0.59) 5.7 (0.45)* 
356 530 


Note: Mean determined froma 6-pointscale, with ratings from “1” indicating “strongly disagree” to “6” 


indicating “strongly agree.” 


tFor reporting purposes, negatively worded items were reverse coded. 
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Appendix B. Fidelity of Implementation Findings by 
Indicator 


The following tables detail findings for each of the 16 indicators included in the four key 
components of the Urban Growth Opportunity (UGO) Fidelity of Implementation (FOD) matrix. 
Table B-1 is only for the mentors who were new each year. All other tables are for the entire 
sample of mentors participating in the project. Each section begins with a narrative description 
of the indicators under each component and concludes with a summary of results for the 
component. 


FOI Findings for Key Component 1, Mentor Recruitment and Assignment 


The Alaska Department of Education and Early Development (AK DEED) recommends a 
minimum qualification for becoming an Alaska Statewide Mentor Project (ASMP) mentor of at 
least eight years of teaching experience in Alaska. We set this as our adequate bar. Mentors 
scored low if they had fewer than eight years of experience teaching in Alaska, and they scored 
ideal if they had more than eight years of such teaching experience. We only assessed this 
indicator the year the mentor was hired. 


In the ASMP model, mentors are fully released from classroom responsibilities, allowing them 
to dedicate their full-time equivalent (FTE) to mentoring. We set this as our bar. Mentors scored 
low if their mentoring occurred in addition to teaching, as this would not afford them the 
opportunity to work with early career teachers (ECTs) during the ECTs’ school day. We scored 
them adequate if their mentoring occurred in addition to other, non-teaching duties in the 
district. We reasoned that as some UGO mentors were district employees, districts might assign 
other duties to them that mentors could address around the schedules they established for 
interacting with their ECTs. We assessed this indicator every year. 


AK DEED assigns a full-time mentor a caseload ofno more than 15 ECTs to provide sufficient 
time for each mentor to spend with their assigned ECTs. We set this as our ideal bar. We scored 
full-time mentors low if they had more than 15 ECTs, because with too many ECTs they would 
be unable to spend sufficient time with each of them. We scored full-time mentors adequate if 
they had fewer than 12 ECTs. With too few ECTs, mentors could spend more than the model’s 
recommended time with each ECT. We assessed this indicator every year. 


The following four tables detail project findings for the three indicators included in the first key 
component of the UGO FOI matrix. Table B-1 summarizes fidelity indicators for the component. 
Tables B-1 through B-3 summarize each indicator across the three years in which fidelity was 
measured. Table B-4 summarizes results for the component. 
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Table B-1. UGO FO! matrix and findings forkey component 1 mentor recruitment/assignment: 
Alaska teaching experience 


Indicator 1.1: AK teaching experience Percentage (n) 

Operational definition: 8 years Y1, 2012-13 Y2,2013-14 3, 2014-15 

Assessed: First-year mentors (n = 10) (n = 9) (n = 2) 

0. Low: Less than 8 years of experience teaching in 0% (0) 41% (1) 0% (0) 
Alaska : : : 

1. Adequate: 8 years of experience teaching in 0% (0) 0% (0) 0% (0) 
Alaska 7 . ° 

2. Ideal: More than 8 years of experience teaching in 
Aiseke y P g 100% (10) 89% (8) 100% (2) 


Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 
Sources: Mentor profile. 


100% (10) 89% (8) 100% (2) 


Table B-2. UGO FO! matrix and findings for key component 1 mentor recruitment/assignment: 
mentoring time 


Indicator 1.2: Mentoring time Percentage (n) 

Operational definition: Full-release Y1, 2012-13 Y2, 2013-14 Y3, 2014-15 
Assessed: All mentors (n= 10) (n = 18) (n = 17) 
0. Low: Mentoring occurs in addition to teaching 0% (0) 0% (0) 0% (0) 


1. Adequate: Mentoring occurs in addition to other, 
non-teaching, duties in the district 
2. Ideal: Mentoring occurs with no other 
responsibilities in district 
Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 
Sources: Mentor profile; interviews. 


0% (0) 0% (0) 12% (2) 
100% (10) 100% (18) 88% (15) 


100% (10) 100% (18) 100% (17) 


Table B-3. UGO FO! matrix and findings for key component 1 mentor recruitment/assignment: 
caseload 
Indicator 1.3: Caseload Percentage (n) 
Operational definition: 1.0 FTE: 12-15 ECTs 
0.5 FTE: 6-7 ECTs 


0.375 FTE: 4-5 ECTs Y1, 2012-13 Y2, 2013-14 Y3, 2014-15 

Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Has a caseload above 15 (if 1.0 FTE), above 0% (0) 0% (0) 12% (2) 

8 (if 0.5 FTE), or above 5 (if 0.375 FTE) 
1. Adequate: Has a caseload of less than 12 (if 1.0 50% (5) 6% (1) 6% (1) 

FTE), less than 6 (if 0.5 FTE), orless than 4 (if 

0.375 FTE) 
2. Ideal: Has a caseload of 12-15 (if 1.0 FTE), 6-7 if 50% (5) 94% (17) 82% (14) 


(0.5 FTE), or 4-5 (if 0.375 FTE) 


Full-sample fidelity score 6 0 0 
Percentage of mentors with fidelity score of 1 or 2 aaa ees) ue 


Sources: Contactlog by calendar week; ASMP dashboard; mentor interview. 
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Table B-4. Fidelity of key component: mentor recruitment/assignment 


Year 1 Year 2 Year 3 
Average percent of mentors with a score of 1 or 2 100% 96% 96% 
across indicators 
Average of the three indicators is at least 70 percent Yes Yes Yes 
of mentors scoring 1 or 2 
At least 51 percent of mentors score 1 or 2 Yes Yes Yes 
Meets fidelity threshold Yes Yes Yes 


Key Component 2: Mentor Participation in Professional Development 


AK DEED expects mentors to fully participate in professional development events. We set 
attending at least 75 percent of scheduled sessions as our ideal bar for Orientation, Wrap Up, 
NTC Academy, and ASMP training. We used this criterion based on the amount of material 
covered; the importance of engaging with the material, trainers, and other mentors; and the 
sheer size of Alaska. It can be difficult to make up missed sessions. Therefore, missing any time 
during the in-person training sessions is highly discouraged.* We scored mentors adequate if 
they attended 50 to 74 percent of the scheduled training, and low if they attended less than 50 
percent. We assessed the indicators for Orientation, Wrap Up, and ASMP training every year. 
We assessed the indicator for NTC Academy in mentors’ first and second year of mentoring 
only. 


AK DEED scheduled approximately 17 Friday Forums per year. Generally they occurred every 
other Friday from September through April, excluding holiday breaks. We set ideal attendance 
to participating in at least 10 sessions per year (including the in-person training during 
Academy week). Mentors scored adequate if they participated in eight to nine sessions and low if 
they participated in fewer than eight sessions. We assessed this indicator every year. 


AK DEED expects coaches and mentors to communicate at least twice a month. Since this is 
similar to the frequency of Friday Forums, we set ideal participation to at least 10 monthly 
conversations per year. Mentors scored adequate if they had eight to nine months with at least 
one documented conversation and low if they had fewer than eight months with one 
documented conversation. We assessed this indicator every year. 


AK DEED expects coaches to shadow first-year mentors twice a year and returning mentors 
once a year. For first-year mentors, we set ideal to participating in two shadowing experiences, 
adequate to participating in one shadowing experience, and Jow to not participating in any 
shadowing experiences. For all other mentors we set ideal to participating in one shadowing 
experiences and low to not participating in any shadowing experiences. We assessed this 
indicator every year. 


3 Mentors could, and did, makeup missed trainings by meeting with their coach one-on-one to review 
the materials. 
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AK DEED expects coaches to guide mentors through a reflective process using a set of tools 
that includes an Individualized Learning Plan (ILP), Mid-Year Gauge (MYG), Mentor 
Collaborative Log, and Personal Growth Reflection (PGR). We set ideal implementation 
criteria to mentors participating in at least four reflective events with their coaches (as 
evidenced by four completed tools). They received a score of adequate if they had two or three 
completed tools and low if they had one or no completed tools. We assessed this indicator every 
year. 


Note: Coaches collect these reflective tools and their documentation of coaching conversations 
and shadows in the Mentor Accountability and Growth Assessment (MAGA) folder. Our 
measurement of coaching comes almost entirely from this documentation. Missing 
documentation does not necessarily indicate that a coaching event did not occur, but rather that 
some coaches were better at documenting activities than others. Thus, variation across mentors 
in their participation in coaching activities may be more of a reflection of their coach’s ability to 
document those activities. 


The following nine tables detail project findings for the eight indicators included in the second 
key component of the UGO FOI matrix. Tables B-5 through B-12 summarize each indicator 
across the three years and Table B-13 summarizes findings for the component. 


Table B-5. UGO FO! matrix and findings forkey component 2 mentor participation in 
professional development: orientation 


Indicator 2.1: Orientation Percentage (n) 
Operational definition: Fully attends as scheduled Y1, 2012-13 Y2,2013-14 Y3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor attends less than 50 percent of 5 p 6 
scheduled orientation oe NS) miele) ett) 
1. Adequate: Adequate: Mentor attends 50-74 0% (0) 0% (0) 0% (0) 
(0) (0) (0) 


percent of scheduled orientation 
2. Ideal: Mentor attends at least 75 percent of 70% (7) 89% (16) 94% (16) 
scheduled orientation 
Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Sources: ASMP dashboard; researcher participation; ASMP and district coordinator reports. 


70% (7) 89% (16 94% (16) 
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Table B-6. UGO FOI! matrix and findings for key component 2 mentor participation in 
professional development: Wrap Up 


Indicator 2.2: Wrap Up Percentage (n) 

Operational definition: Fully attends as scheduled Y1, 2012-13 Y2, 2013-14 Y3, 2014-15 

Assessed: All mentors (n = 10) (n= 18) (n= 17) 

0. Low: Mentor attends less than 50 percent of Wrap 10% (1) 6% (1) 6% (1) 
Up 

1. Adequate: Mentor attends 50-74 percent of Wrap 0% (0) 0% (0) 0% (0) 
Up 

2. Ideal: Mentor attends at least 75 percent of Wrap 90% (9) 94% (17) 94% (16) 

Up 

Full-sample fidelity score 90% (9) 94% (18) 94% (16) 


Percentage of mentors with fidelity score of 1 or 2 
Source: ASMP dashboard; researcher participation; ASMP and district coordinator reports. 


Table B-7. UGO FOI matrix and findings for key component 2 mentor participation in 
professional development: Academy 


Indicator 2.3: Academy Percentage (n) 

Operational definition: Fully attends as scheduled Y1, 2012-13 Y2,2013-14 Y3, 2014-15 

Assessed: First- and second-year mentors (n = 10) (n = 18) (n = 8) 

0. Low: Mentor attends less than 50 percent of 5 Fs ‘ 
Academy 0% (0) 0% (0) 0% (0) 

1. Adequate: Mentor attends 50-74 percent of 0% (0) 0% (0) 0% (0) 
Academy e . 

2. Ideal: Mentor attends at least 75 percent of 6 é 6 

Academy 100% (10) 100% (18) 100% (8) 


Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Sourcse: ASMP dashboard; researcher participation; ASMP and district coordinator reports. 


100% (10) 100% (18) 100% (8) 


Table B-8. UGO FOI! matrix and findings for key component 2 mentor participation in 
professional development: ASMP training 


Indicator 2.4: ASMP training Percentage (n) 

Operational definition: Fully attends as scheduled Y1, 2012-13 Y2,2013-14 Y3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. penicad attends less than 50 percent of ASMP 0% (0) 0% (0) 0% (0) 
1. hha Mentor attends 50—74 percent of ASMP 0% (0) 0% (0) 0% (0) 
2. Ideal: Mentor attends at least 75 percent of ASMP 


0, 0, 0, 
iaiine 100% (10) 100% (18) 100% (17) 


Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Sources: ASMP dashboard; researcher participation; ASMP and district coordinator reports. 


100% (10) 100% (18) 100% (17) 
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Table B-9. UGO FOI matrix and findings for key component 2 mentor participation in 
professional development: Friday Forums 


Indicator 2.5: Friday Forums Percentage (n) 

Operational definition: Attends at least 10 forums Y1, 2012-13 Y2,2013-14 Y3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor attends 0-7 Friday Forums 0% (0) 0% (0) 0% (0) 

1. Adequate: Mentor attends 8-9 Friday Forums 0% (0) 0% (0) 0% (0) 

2. Ideal: Mentor attends 10 or more Friday Forums 100% (10) 100% (18) 100% (17) 


Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Sources: ASMP dashboard; co-facilitator reports. 


100% (10) 100% (18) 100% (17) 


Table B-10. UGO FO! matrix and findings for key component 2 mentor participation in 
professional development: mentor communicates with coach (Coaching Conversations 
Indicator 2.6: At least monthly communication with Percentage (n) 
ASMP coach (Coaching Conversations) 
Operational definition: 10 regularly scheduled 
through academic year Y1, 2012-13 Y2,2013-14 3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor has fewer than 8 months with at least 


one communication 0% (0) 0% (0) 0% (0) 
1. Adequate: Mentor has 8-10 months with at least 

one communication 60% (6) 27% (5) 53% (9) 
2. Ideal: Mentor has atleast 10 months with at least 40% (4) 72% (13) 47% (8) 


one communication 
Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Source: Mentor accountability and growth assessment system (MAGA). 


100% (10) 100% (18) 100% (17) 


Table B-11. UGO FO! matrix and findings for key component 2 mentor participation in 
professional development: shadowing by coach 


Indicator 2.7: Shadowing Percentage (n) 
Operational definition: Twice a year (first year) and at 

least once a year (second year or more) Y1, 2012-13. Y2, 2013-14 = Y3, 2014-15 
Assessed: All mentors (n = 10) (n = 18) (n = 17) 
0. Low: Mentor participates in 0 shadowing activities 0% (0) 5% (1) 6% (1) 
1. Adequate: Adequate: Mentor participates in 50% (5) 6% (1) 0% (0) 


one shadowing activity (first-year mentors only) 
2. Ideal: Mentor participates in two or more 
shadowing activities in the first year and one or 50% (5) 89% (16) 94% (16) 
more times in the second year and beyond 
Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Sources: Mentor accountability and growth assessment system (MAGA); interviews. 


100% (10) 95% (17) 94% (16) 
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Table B-12. UGO FO! matrix and findings for key component 2 mentor participation in 
professional development: mentor coached using formative assessment tools by coach 


Indicator 2.8: Mentor is coached using formative Percentage (n) 

assessment 
Operational definition: Mentor is coached using 

ASMP tools for coaches Y1, 2012-13 Y2,2013-14 3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 


0. Low: Mentor is coached using 0-1 ASMP tools 
(e.g., Individual Learning Plan (ILP), Mid-Year 


Growth (MYG), Collaborative Assessment Logs 0% (0) 0% (0) 0% (0) 
(CALs) or Professional Growth Reflection (PGR) 
1. pee corer glial is coached using two to three 90% (9) 67% (12) 29% (5) 
2. gior aia is coached using four or more ASMP 10% (1) 33% (6) 71% (12) 


Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Source: Mentor accountability and growth assessment system (MAGA). 


100% (10) 100% (18) 100% (17) 


Table B-13. Fidelity of key component 2 mentor participation in professional development 


Year 1 Year 2 Year 3 
Average percentage of mentors with a score of 1 or 2 95% 97% 98% 
across indicators 
Average of the eight indicators is at least 70 percent Yes Yes Yes 
of mentors scoring 1 or 2 
At Least 51 percent of mentors score 1 or 2 Yes Yes Yes 
Meets fidelity threshold Yes Yes Yes 


Key Component 3: Mentor Interacts With ECTs 


AK DEED expects mentors to communicate weekly with ECTS via phone, email, or Skype, fora 
total of approximately 28 communications between mid-September, after most hiring by 
districts is complete, and mid-May, when mentors participate in Wrap Up, excluding holidays 
and breaks. We set ideal to mentors having at least 22 weekly communications with at least 80 
percent of their ECTs. Mentors received an adequate score if they had 22 weekly communications 
with 50 to 79 percent of their ECTs, and a low score if they had the same number of 
communications with fewer than 50 percent of their ECT's. We assessed this indicator for all 
mentors every year. We excluded from the calculations any ECT that we knew had taken some 
type of leave (e.g., medical) during the school year and, as a result, did not receive 22 weekly 
contacts from their mentor. 


AK DEED expects mentors to engage in at least 3.5 hours of face-to-face contact with each ECT 
per month. We set ideal to mentors having at least six months throughout the year when their 
face-to-face interactions totaled three hours with at least 80 percent of their ECTs. Mentors 
received an adequate score if they had six months through the year when their face-to-face 
interactions totaled three hours with 50 to 79 percent of their ECT's. They obtained a low score if 
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they had accumulated three hours of face-to-face contact for six months with fewer than 50 
percent of their ECTs. We assessed this indicator for all mentors every year. We excluded from 
the calculations any ECT that we knew had taken some type of leave (e.g., medical) during the 
school year and, as a result, did not receive six months of face-to-face visits totaling three anda 
half hours. 


The following three tables detail project findings for the two indicators included in the third key 
component of the UGO FOI matrix. Tables B-14 through B-15 summarize each indicator across 
the three years and Table B-16 summarizes findings for the component. 


Table B-14. UGO FOI matrix and findings for key component 3 mentor interacts with ECTs: 
weekly communication 


Indicator 3.1: Weekly communication Percentage (n) 
Operational definition: 28 per year (e.g., email, 
phone, Skype) with each ECT Y1, 2012-13 Y2,2013-14 3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor has at least 22 weekly 
communications with fewer than 50 percent of 0% (0) 0% (0) 0% (0) 
ECTs 


1. Adequate: Mentor has at least 22 weekly 
communications with 50-79 percent of ECTs 
2. Ideal: Mentor has at least 22 weekly 
communications with at least 80 percent of ECTs 
Full-sample fidelity score 
Percentage of mentors with fidelity score of 1 or 2 
Source: Contactlog by calendar week. 


0% (0) 0% (0) 0% (0) 
100% (10) 100% (18) 100% (17) 


100% (10) 100% (18) 100% (17) 


Table B-15. UGO FO! matrix and findings for key component 3 mentor interaction with ECTs: 
face-to-face interactions 
Indicator 3.2: Regular face-to-face interactions Percentage (n) 
Operational definition: 3.5 hours each month in face- 
to-face interactions, distributed through the year 


with each ECT Y1, 2012-13 Y2, 2013-14 3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor has at least six months through the 

year when face-to-face interactions total three 30% (3) 28% (5) 18% (3) 


hours for 50 percent or more of ECTs 
1. Adequate: Mentor has at least six months through 
the year when face-to-face interactions total three 30% (3) 28% (5) 18% (3) 
hours for 50-79 percent of ECTs 
2. Ideal: Mentor has atleast six months through the 
year when face-to-face interactions total three 40% (4) 44% (8) 65% (11) 
hours for 80 percent or more of ECTs 
Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 
Source: Contactlog by calendar week. 


70% (7) 72% (13) 83% (14) 
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Table B-16. Fidelity of key component 3 mentor interacts with ECTs 


Year 1 Year 2 Year 3 
Average percentage of mentors with a score of 1 or 2 85% 86% 92% 
across indicators 
Average of the two indicators is at least 70 percent of Yes Yes Yes 
mentors scoring 1 or 2 
At least 51 percent of mentors score 1 or 2 Yes Yes Yes 
Meets fidelity threshold Yes Yes Yes 


Key Component 4: Mentor Use of Formative Assessment with ECTs 


AK DEED expects mentors to document their weekly contacts with ECTs using the 
Collaborative Assessment Log (CALs). We set ideal mentors having at least 22 completed CALs 
in at least 80 percent of their ECTs’ folders. Mentors received an adequate score if they had 22 
completed CALs in 50 to 79 percent of their ECTs folders and a low score if they had the same 
number of completed CALs in less than 50 percent of their ECTs folders. In assessing this 
indicator, we reviewed all mentors’ ECT folders each year. We considered CALs completed if 
mentors completed two of the four quadrants on the form. Again, we excluded from the 
calculations any ECT that we knew had taken some type of leave (e.g., medical) during the 
school year and, as a result, did not have 22 completed CALs in their folder. 


AK DEED expects mentors to use a variety of NTC/ASMP formative assessment tools with 
their ECTs. We set ideal to mentors having used at least four tools across the two semesters (e.g, 
one tool in fall and three in spring or two in fall and two in spring) with at least 80 percent of 
their ECTs. Mentors received an adequate score if they used four tools across the two semesters 
with 50 to 79 percent of their ECTs and a low score if they used the same number with less than 
50 percent of their ECTs. In assessing this indicator, we reviewed all mentors’ ECT folders each 
year. We counted any tool documented in folders (e.g., a seating chart drawn ona scrap of 
paper that was not a formal data collection tool) and allowed multiple uses of the same tool to 
count as multiple instances of tool use. Again, we excluded from the calculations any ECT that 
we knew had taken some type of leave (e.g., medical) during the school year and, as a result, 
did not have four tools used across two semesters. 


AK DEED expects mentors to guide ECTs through a reflective process using a set of tools 
comprised of an Individualized Learning Plan (ILP), Mid-Year Gauge (MYG), and Personal 
Growth Reflection (PGR). We set ideal implementation criteria to mentors using an ILP and at 
least one additional reflective practice tool with at least 80 percent of their ECTs. Mentors 
received an adequate score if they used the ILP and at least one additional tool with 50 to 79 
percent of their ECTs; they scored low if they used they used only the ILP or the ILP and at least 
one other tool with less than 50 percent of their ECTs. In assessing this indicator, we reviewed 
ECT foldersevery year. Again, we excluded from the calculations any ECT that we knew had 
taken some type of leave (e.g., medical) during the school year and, as a result, did not have the 
ILP and/or other reflective practice tools documented in their folder. 
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The following four tables detail project findings for the three indicators included in the fourth 
key component of the UGO FOI matrix. Tables B-17 through B-19 summarize each indicator 
across the three years and Table B-20 summarizes findings for the component. 


Table B-17. UGO FOI matrix and findings for key component 4 mentor use of formative 
assessment with ECTs: collaborative assessment logs 


Indicator 4.1: Collaborative Assessment Logs (CALs) Percentage (n) 
Operational definition: 28 CALs used per year with 
each ECT Y1, 2012-13 Y2,2013-14 3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n= 17) 
0. Low: Mentor has completed at least 22 CALs each 
with fewer than 50 percent of ECTs 0% (0) 0% (0) 0% (0) 
1. Adequate: Mentor has completed at least 22 
CALS each with 50-79 percent of ECTs 0% (0) 0% (0) 0% (0) 
2. Ideal: Mentor has completed at least 22 CALs 
each with 80 percent or more of ECTs 100% (10) 100% (18) 100% (17) 


Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 100% (10) 100% (18) 100% (17) 


Source: Mentor folders on each ECT. 


Table B-18. UGO FO! matrix and findings for key component 4 mentor use of formative 
assessment with ECTs: various tools 
Indicator 4.2: Various formative assessment tools Percentage (n) 
regularly used to collect data 
Operational definition: Uses a variety of formative 
assessment tools/ strategies with ECTs during 
the year (in addition to the CAL) with each ECT Y1, 2012-13 Y2,2013-14 Y3, 2014-15 
Assessed: All mentors (n = 10) (n= 18) (n = 17) 
0. Low: Mentor uses at least four tools each, across 
ee semesters, with fewer than 50 percent of 40% (4) 22% (4) 18% (3) 
ECTs 
1. Adequate: Mentor uses at least four tools each, 


across both semesters, with 50-79 percent of 10% (1) 17% (3) 35% (6) 
ECTs 


2. Ideal: Mentor uses at least four tools each, across 
both semesters, with 80 percent or more of ECTs 
Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 
Source: Mentor folders on each ECT. 


50% (5) 61% (11) AT% (8) 


60% (6) 78% (14) 82% (14) 
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Table B-19. UGO FO! matrix and findings for key component 4 mentor use of formative 


assessment with ECTs: reflective practice tools 
Indicator 4.3: Reflective Practice 
Operational definition: Individual Learning Plan 
(ILP), Mid-Year Review, and Professional Growth 
Reflection used with each ECT 
Assessed: All mentors 
0. Low: Mentor uses no reflective practice tools, only 
ILP or ILP and at least one other tool with fewer 
than 50 percent of ECTs 
1. Adequate: Mentor uses ILP and at least one 
additional reflective practice tool each with 50—79 
percent of ECTs 
2. Ideal: Mentor uses ILP and at least one additional 
reflective practice tool each with 80 percent or 
more of ECTs 
Full-sample fidelity score 
(Percentage of mentors with fidelity score of 1 or 2) 
Source: Mentor folders on each ECT. 


Percentage (n) 


Y1, 2012-13 »=9Y2, 2013-14 Y3, 2014-15 


(n = 10) (n= 18) (n = 17) 
0% (0) 0% (0) 0% (0) 
0% (0) 6% (1) 0% (0) 


100% (10) 94% (17) 100% (17) 


100% (10) 100% (18) 100% (17) 


Table B-20. Fidelity of key component 4 mentor use of formative assessment with ECTs 


Average percentage of mentors with a score of 1 or 2 
across indicators 

Average of the three indicators is at least 70 percent 
of mentors scoring 1 or 2 

At least 51 percent of mentors score 1 or 2 


Meets fidelity threshold 
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Year 1 Year 2 Year 3 
87% 93% 94% 
Yes Yes Yes 
Yes Yes Yes 
Yes Yes Yes 


Appendix C. Fidelity of Implementation Matrix and 
Findings by Component and Year 


This document shows Fidelity of Implementation (FOI) tables and results summarized by 
component and year. This information was requested by the NEi3 evaluation. 


Table C-1. Description of key components 


Planned intervention activity List of key indicators for each key component 


Mentor recruitment/assignment 1. AK teaching experience (first-year mentors only) 
2. Mentoring time (all mentors every year) 
3. Caseload (all mentors every year) 
Mentor participation in 1. Orientation (all mentors every year) 
professional development 2. Wrap Up (all mentors every year) 
3. Academy (first- and second-year mentors only) 
4. ASMP training (all mentors every year) 
5. Friday forums (all mentors every year) 
6. Communication with ASMP coach (all mentors every year) 
7. Shadowed by coach (all mentors every year) 
8. Coached using formative assessment system (FAS) (all mentors 
every year) 
Mentor interaction with early 1. Weekly communication (all mentors every year) 
career teachers (ECTs) 2. Face-to-face interactions (all mentors every year) 
Mentor use of formative 1. Collaborative Assessment Logs (CALs) (all mentors every year) 
assessment tools with ECTs 2. Various FAS/ASMP and other tools (all mentors every year) 
3. Reflective practice tools (all mentors every year) 
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Table C-2. Findings from evaluator study of implementation: Implementation Year 1 (August 


2012—May 2013) 


Intervention 
components 


Mentor 
recruitment/ 
assignment 


participation 
in 

professional 
development 


108 Education Northwest 


Implementation 


measure 


Representative- 


ness 


level threshold 


Component 
for FOI 


Mentors: 


Have eight years 
teaching in AK 
Are fully released 
from teaching 
Have caseloads 
of 12-15 ECTs 
(1.0 FTE) 


Orientation: fully 
attend as 
scheduled 
Wrap Up: fully 
attend as 
scheduled 


Academy: first- 
and second-year 
mentors fully 
attend as 
scheduled 


ASMP: fully 
attend as 
scheduled 
Friday Forums: 
attend at least 10 
Communication 
with ASMP 
coach: at least 10 
throughout 
academic year 
Shadowed by 
coach: first-year 
mentors, two per 
year; second- 
year (or more) 
mentors, at least 
One per year 
Coached using 
FAS: at least four 
tools used 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


Component 


level 
fidelity score 


Implemented 
with fidelity 


100 percent of | Yes 


mentors had 
average 
scores of at 
least 
adequate, and 
100 percent of 
mentors 
scored at 
least 
adequate for 
each indicator 


95 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
70 percent of 
mentors 
scored at 
least 
adequate for 
each indicator 


Implementation 


Intervention 
measure 


components 


Mentor 


interaction 
with ECTs 


Mentors use | 3 
of formative 
assessment 
tools with 
ECTs 


x 
[oe 
Eo 
© N 
Nn 


> 


= 


Representative- 


ness 


Component 
level threshold 


Weekly 
communication, 
28 per year 
Face-to-face 
interactions, 3.5 
hours per month 


Collaborative 
Assessment Log, 
28 per year 
Various formative 
assessment 
tools, at least four 
tools used across 
both semesters 
with at least 80 
percent of ECTs 
Reflective 
practice tools, ILP 
and at least one 
additional tool 
used with at least 
80 percent of 
ECTs 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


Component 
fidelity score 


level 


85 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
70 percent of 
mentors 
scored at 
least 
adequate for 
each indicator 


80 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
60 percent of 
mentors 
scored at 
least 
adequate for 
each indicator 


Implemented 
with fidelity 
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Table C-3. NEi3 findings from evaluator study of implementation: Implementation Year 2 


(August 2013—May 2014) 


Implementation 


measure 
Representative- 


Intervention 
ness 


components 


Mentor 


assignment 


Mentor 
participation 
in 
professional 
development 


110 Education Northwest 


level threshold 


Component 
for FOI 


Mentors: 
recruitment/ 1. 


Have eight 
years teaching 
in Alaska 

Are fully 
released from 
teaching 

Have 
caseloads of 
12-15 ECTs 
(1.0 FTE) 


Orientation: 
fully attend as 
scheduled 
Wrap Up: fully 
attend as 
scheduled 
Academy: First- 
and second- 
year mentors 
fully attend as 
scheduled 
ASMP: fully 
attend as 
scheduled 
Friday Forums: 
attend at least 
10 
Communication 
with ASMP 
coach: at least 
10 throughout 
academic year 
Shadowed by 


coach: first year 


mentors, two 
per year; 
second year or 
more mentors, 


at least one per 


year 
Coached using 
formative 
assessment 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, and 
at least 51 
percent of 
mentors 
scored “1” 0n 
each indictor” 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, and 
at least 51 
percent of 
mentors 
scored “1” on 
each indictor’” 


Component 
fidelity score 


level 


96 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
89 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


97 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
89 percent of 
mentors 
scored at least 
adequate for 
each indicator 


Implemented 
with fidelity 


Intervention 
components 


Mentor 


interaction 
with ECTs 


Mentors use 
of formative 
assessment 
tools with 
ECTs 


Implementation 


Representative- 


Component 
level threshold 


tools: at least 
four tools used 


Weekly 
communication, 
28 per year 
Face-to-face 
interactions, 3.5 
hours per 
month 


CALs, 28 per 
year 

Various 
formative 
assessment 
tools, at least 
four tools used 
across both 
semesters with 
at least 80 
percent of 
ECTs 
Reflective 
practice tools, 
ILP and at least 
one additional 
tool used with 
at least 80 
percent of 
ECTs 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, and 
at least 
51percent of 
mentors 
scored “1” on 
each indictor’” 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, and 
at least 51 
percent of 
mentors 
scored “1” 0n 
each indictor” 


Component 
fidelity score 


level 


86 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
72 percent of 
mentors 
scored atleast 
Adequate for 
each indicator 


93 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
78 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


Implemented 
with fidelity 
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Table C-4. NEi3 findings from evaluator study of implementation: Implementation Year 3 
(August 2014—May 2015) 


Intervention 
components 


Mentor 
recruitment/ 
assignment 


Mentor 
participation 
in 
professional 
development 
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Implementation 


measure 


Representative- 


ness 


level threshold 


Component 
for FOI 


Mentors: 


Have eight years 
teaching in 
Alaska 

Are fully released 
from teaching 


Have caseloads 
of 12-15 ECTs 
(1.0 FTE) 


Orientation: fully 
attend as 
scheduled 
Wrap Up: fully 
attend as 
scheduled 


Academy: first- 


and second-year 
mentors fully 
attend as 
scheduled 


ASMP: fully 
attend as 
scheduled 


Friday Forums: 
attend at least 10 


Communication 
with ASMP 
coach: at least 10 
throughout 
academic year 


Shadowed by 
coach: first-year 
mentors, two per 
year; second- 
year (or more) 
mentors, at least 
one per year 


Coached using 
formative 
assessment tools 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


Component 
fidelity score 


level 


96 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
88 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


98 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
94 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


Implemented 
with fidelity 


Intervention 
components 


Mentor 


interaction 
with ECTs 


Mentors use 
of formative 
assessment 
tools with 
ECTs 


Implementation 


measure 


Representative- 


Component 
level threshold 


at least four tools 
used 


Weekly 
communication, 
28 per year 
Face-to-face 
interactions, 3.5 
hours per month 


CALs, 28 per 
year 

Various formative 
assessment 
tools, at least four 
tools used across 
both semesters 
with at least 80 
percent of ECTs 
Reflective 
practice tools, ILP 
and at least one 
additional tool 
used with at least 
80 percent of 
ECTs 


Criteria for 
implemented 
with fidelity 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


70 percent of 
mentors had 
an average of 
at least “1” 
across all 
indicators, 
and at least 
51 percent of 
mentors 
scored “1” on 
each indictor’ 


Component 
fidelity score 


level 


92 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
83 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


94 percent of 
mentors had 
average 
scores of at 
least 
adequate, and 
82 percent of 
mentors 
scored atleast 
adequate for 
each indicator 


Implemented 
with fidelity 
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Appendix D. UGO Consort Charts for All Contrasts 


Acronyms 

ASD Anchorage School District 
ATI Alaska Teacher Identifier 
C1, C2,C3 Cohort 1, Cohort 2, Cohort 3 
DOD Department of Defense 
Br Early Career Teacher 

PE Physical Education 

SPED Special Education 

Y1, Y2 Year 1, Year 2 


Consort Charts for Teacher Attrition 


Outcome Aa, Contrast 1, Teacher Attrition (all cohorts pooled, confirmato 


ECTs initially considered for recruitment 
Determined to be ineligible for stud 
Approached for consent 
Declined 
ECTs consented and randomly assigned 


Began study beginning of Y1 
Determined to be ineligible for stud 

Eligible for Outcome A, Contrast 1, Teacher Attrition 

Total Missing Outcome Data 

Missing ATI 

Missing Outcome Data 

Dropped from Model 

Estimation Sample 

Attrition 

Differential Attrition 

Overall Attrition 
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All Cohorts 


605 


556 
Treatment 
286 


286 


267 


—_ —_ 
(oe) -|/0 
@|O 
alee 
os |os 


Control 

270 
1 
269 
17 

1 

8 

8 


Outcome Ab, Contrast 1, Teacher Attrition (Cohort 1 only, exploratory) 


Cohort 1 
Cohort 1 ECTs initially considered for recruitment 205 
Determined to be ineligible for study 25 
Approached for consent 180 
Declined 24 
Cohort 1 ECTs consented and randomly assigned 156 
Control 
Began Study Beginning of Y1 73 
Determined to be ineligible for study (a | 
Eligible for Outcome A, Contrast 1, Teacher Attrition 73 
Total Missing Outcome Data Po 
Missing ATI pe 
Missing Outcome Data Po 
Dropped from Model Po 
Estimation Sample 70 
Attrition =  ] 


Differential Attrition 
Overall Attrition 
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Consort Charts for Instructional Practice 


Outcome Ba, Contrast 1, Emotional Support (Year 2, Cohorts 2 and 3 pooled, confirmatory) 
Outcome Ca, Contrast 1, Classroom Organization (Year 2, Cohorts 2 and 3 pooled, confirmatory) 
Outcome Da, Contrast 1, Instructional Support (Year 2, Cohorts 2 and 3 pooled, confirmato 


Cohorts 2&3 
Control 

ECTs randomly assigned to instructional recordings prior 86 
to applying a priori exclusion criteria 
Total Excluded A Priori 22 

PreK ECTs ——— al 3 

ECTs providing instruction in language other than 

English or Spanish 

DOD base ECTs (in ASD 3 

SPED teachers (in ASD 10 

Baseline video not scorable (technical difficulties 1 
ECTs randomly assigned, with replacement, to ea 
instructional recording with scorable baseline recordings, 70 64 
Fall Y1 
Total ECTs missing Year 2 outcome data 18 

Dropped before Y2 (did not want to be videotaped, 

grant funded through not renewed, left district/moved ss) 9 

out of state, non-retained (district reasons), resigned) 

Transferred to special population (Y2 or C3) (juvenile 

justice, expelled, special education, religion, mixed oe 8 

student/adult classroom, PE, DOD school 

Error 0 

Opted out of video recordings after consenting 1 

Maternity leave SS sae 1 

Could not attribute score to ECT 0 
Estimation Sample 46 
Attrition 28.1% 
Overall Attrition 27.6% 
Differential Attrition 1.0% 
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Outcome Bb, Contrast 1, Emotional Support (Year 1, Cohorts 2 and 3 pooled, exploratory) 
Outcome Cb, Contrast 1, Classroom Organization (Year 1, Cohorts 2 and 3 pooled, exploratory) 
Outcome Db, Contrast 1, Instructional Support (Year 1, Cohorts 2 and 3 pooled, explorato 
Cohorts 2&3 
Treatment Control 


86 


22 
3 


ECTs randomly assigned to instructional recordings 
prior to applying a priori exclusion criteria 
Total Excluded A Priori 
PreK ECTs 
ECTs providing instruction in language other than 
English or Spanish 
DOD base ECTs (in ASD 
SPED teachers (in ASD 
Baseline video not scorable (technical difficulties 
ECTs randomly assigned, with replacement, to 
instructional recording with scorable baseline 
recordings, Fall Y1 (Eligible for Outcomes BCD 
Total ECTs missing Year 1 outcome data 
Not recorded in Year 2, and Y1 videos not sent for 
scoring (dropped after Y1, moved into non- 
recordable classroom, maternity leave) 
Error 
Opted out of video recordings after consenting 
Dropped study during Y1 
Spring Y1 videos not scorable 
Could not attribute score to ECT 
Estimation Sample 
Attrition 25.7% 29.7% 
Overall Attrition 27.6% 
Differential Attrition 4.0% 


Ny} © 
oO 


ol ass |e sy 

NR] a}o}r]3]/— Oo |ININIn| = = 
rep) 
K 


(ee) 
—_ 
© 


wo 
—_ 
o 
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Consort Charts for Student Achievement 


Outcome Ea, Student Reading Primary Grades Y2 (Cohorts 1 and 2 pooled, confirmatory) 


Cohorts 1 & 2° 

Control 
Level of Random Assignment ae oo | 
Y2 ECTs with students in grades 4-6 eligible for state 26 23 
reading assessment 
Y2 ECTs Dropped During Y2 a Paes 0 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level ————S | 
Y2 ECTs with student rosters collected in spring 23 
Students on Y2, October 1 roster 472 
Students on Y2, October 1 roster with state reading 527 453 


assessment data next spring 
Students missing covariates 
Estimation sample 

Attrition 11.5% 8.3% 
Differential Attrition 
Overall Attrition 10.0% 


& 
Cj 
O|— 

(ee) 

8 

oS 
& 
w 
wo 


Outcome Eb, Contrast 1, Student Reading Primary Grades Y1 (All cohorts pooled, exploratory) 


All Cohorts 
Control 

Level of Random Assignment eS | 
Y1ECTs with students in grades 4-6 eligible for state 34 33 
reading assessment 
Y1ECTs Dropped During Y1 ee (| 0 
Y1ECTs with student rosters collected in spring 33 
Attrition 0% 

Differential Attrition 0% 

Overall Attrition 0% 

Student Level a el 
Students on Y1, October 1 roster 673 633 
Students on Y1, October 1 roster with state reading 653 606 


assessment data next spring 
Students missing covariates 
Estimation Sample 


o 
0010) 
00/01 
o 
NSN 
Oo) 


Attrition 12.6% 9.0% 
Differential Attrition 3.6% 
Overall Attrition 10.9% 


4 Cohort3 is excluded because the state assessment was not administered in spring 2016. 
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Outcome Fa, Student Writing Primary Grades Y2 (Cohorts 1 and 2 pooled, confirmatory) 
Cohorts 1 & 2° 


Control 
Level of Random Assignment SSS 
Y2 ECTs with students in grades 4-6 eligible for state 25 23 
writing assessment 
Y2ECTs Dropped During Y2 ES ti) 0 
Y2 ECTs with student rosters collected in spring 23 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 


Student-level 


a | 
Students on Y2, October 1 roster 499 


Students on Y2, October 1 roster with state writing 


assessment data next spring 495 465 
Students missing covariates 19 
Estimation sample 446 
Attrition 10.6% 
Differential Attrition 0.5% 
Overall Attrition 10.9% 


Outcome Fb, Student Writing Primary Grades Y 1 (All cohorts pooled, exploratory) 


All Cohorts 


Level of Random Assignment 
Y1 ECTs with students in grades 4-6 eligible for state 


writing assessment 


oO 
© 
Ww 
oO 


oO 
© 
[o) 
wo 
HP 


Differential Attrition 


S 
x 


Student-level 
Students on Y1, October 1 roster 1,009 


Students on Y1, October 1 roster with state writing 977 
assessment data next spring 


(ep) 
o 
© 


10.7% 
2.5% 
9.7% 


8.2% 


oO 

OoOInN 

ee Kop) 
[op) 
Ww 
BK 


(ep) 
OIN 
O11; 


Overall Attrition 


° Cohort 3 is excluded because the state assessment was not administered in spring 2016. 
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Outcome Ga, Student Math Primary Grades Y2 (Cohorts 1 and 2 pooled, confirmatory) 
Cohorts 1 & 2° 


Control 
Level of Random Assignment ae eto | 
Y2 ECTs with students in grades 4-6 eligible for state 27 20 
math assessment 
Y2 ECTs Dropped During Y2 Ei 0 
Y2 ECTs with student rosters collected in spring 22 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level 
Students on Y2, October 1 roster 630 443 
Students on Y2, October 1 roster with state math 409 
assessment data next spring 
Students missing covariates 21 
Estimation sample 388 
Attrition 12.4% 
Differential Attrition 2.2% 
Overall Attrition 11.1% 


Outcome Gb, Student Math Primary Grades Y1 (All cohorts pooled, exploratory) 


All Cohorts 
Control 
Level of Random Assignment SS 
Y1ECTs with students in grades 4-6 eligible for state 36 34 
math assessment 
Y1ECTs Dropped During Y1 a (ie) 0 
Y1ECTs with student rosters collected in spring 34 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 


Student-level 
Students on Y1, October 1 roster 677 650 


Students on Y1, October 1 roster with state math 654 626 
assessment data next spring 
=i 


Students missing covariates 46 
Estimation sample 580 
Attrition 10.8% 
Differential Attrition 1.4% 
Overall Attrition 11.5% 


° Cohort 3 is excluded because the state assessment was not administered in spring 2016. 
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Outcome Gc, Student Math Secondary Grades Y2 (Cohorts 1 and 2 pooled, confirmatory) 
Cohorts 1 & 2’ 
Treatment Control 


Level of Random Assignment 
Y2 ECTs with students in grades 7-10 eligible for state 
math assessment 
Y2 ECTs Dropped During Y2 

Overall Attrition 
Student-level aaa 


assessment data next spring 
59 
14.0% 


Differential Attrition 
Overall Attrition 
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Outcome Gd, Student Math Secondary Grades Y11 (All cohorts pooled, exploratory) 


All Cohorts 
Control 
Level of Random Assignment ae | 
Y 1 ECTs with students in grades 7-10 eligible for state 26 26 
math assessment 
Y1ECTs Dropped During Y1 Se aa 0 
Y1ECTs with student rosters collected in spring 26 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level (ee eee | 
Students on Y1, October 1 roster 1,463 
Students on Y1, October 1 roster with state math 
assessment data next spring 1,369 
Missing covariates 93 
Estimation sample 1,276 
Attrition 12.8% 
Differential Attrition 0.2% 
Overall Attrition 12.9% 


7 Cohort 3 is excluded because the state assessment was not administered in spring 2016. 
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Outcome Ha, Caucasian Student Math Secondary Grades Y 1 (All cohorts pooled, exploratory) 


All Cohorts 


jer asa 


Level of Random Assignment 

Y 1 ECTs with Caucasian students in grades 7-10 eligible 
for state math assessment 

Y1ECTs with Caucasian students Dropped DuringY1 | 0 | 
Y1ECTs with with Caucasian students on rosters 


collected in spring 25 
Attrition 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level De | 
Caucasian students on Y1, October 1 roster 
Caucasian students on Y1, October 1 roster with state 760 
math assessment data next spring 
Caucasian students missing covariates 
Estimation Sample 
Attrition 
Differential Attrition 0.4% 
Overall Attrition 12.1% 


Control 


Outcome |, Hispanic Student Math Secondary Grades Y 11 (All cohorts pooled, exploratory) 


All Cohorts 


S| 


Level of Random Assignment 

Y 1 ECTs with Hispanic students in grades 7-10 eligible 
for state math assessment 

Y1 ECTs with Hispanic students Dropped During Y1 | 
Y 1 ECTs with Hispanic students on rosters collected in 


spring ea 
Attrition 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level Ss 
Hispanic students on Y1, October 1 roster 
Hispanic students on Y1, October 1 roster with state 
math assessment data next spring 
Hispanic students missing covariates 
Estimation Sample 
Attrition 
Differential Attrition 2.9% 
Overall Attrition 15.9% 
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Control 


23 
0 


Outcome J, Alaska Native Student Math Secondary Grades Y11 (All cohorts pooled, exploratory) 
All Cohorts 
Control 
Level of Random Assignment 


aes SS | 
Y 1 ECTs with Alaskan Native students in grades 7-10 
aoe 24 25 
eligible for state math assessment 
A. 


Y1 ECTs with Alaskan Native students Dropped During 


0 
Y1 
Y1 ECTs with Alaskan Native students on rosters 25 
collected in spring 
Attrition 0% 
Differential Attrition 0% 
Overall Attrition 0% 
Student-level (ein | 
Alaskan Native students on Y1, October 1 roster 165 
Alaskan Native students on Y1, October 1 roster with 4140 454 
state math assessment data next spring 
Alaskan Native students missing covariates 8 
Estimation Sample 146 
Attrition 11.5% 
Differential Attrition 0.6% 
Overall Attrition 11.8% 


Outcome K, American Indian Student Math Secondary Grades Y 1 (All cohorts pooled, exploratory 


Trx 
| LevelofRandomAssignment Cd 


Level of Random Assignment 
eligible for state math assessment 
Y1 

collected in spring 


Attrition 
Differential Attrition 


. 
pa 

Ss 

~~ 


- SIS neal 
os |os 
“NI “NI 


12 

American Indian students on Y1, October 1 roster with 

2 
16.7% 

2.1% 
17.9% 


Student-level 


—s 

oO 
= = j— 
wlAayAa} oO IO 


se 
oO 


18.8% 


Overall Attrition 
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Outcome L, Two or More Races Student Math Secondary Grades Y1 (All cohorts pooled, exploratory) 


All Cohorts 
| LevelofRandomAssignment 


Level of Random Assignment SSSsaq) 
Y1ECTs with Two or More Races students in grades 7- 
ke 21 
10 eligible for state math assessment 
Y1 CTs with Two or More Races students Dropped a 
During Y 1 
1 


Y1ECTs with Two or More Races students on rosters 2 
collected in spring 


|Studentlevel ss sc Ctti“(‘(‘(‘(‘(‘#$sé‘(wrwrwdsSaWKXWXMh 


| Two or More Races students missing covariates | 
: 


Two or More Races students on Y1, October 1 roster 
d : 109 
with state math assessment data next spring 


Outcome M, Districts without Formal Mentoring Programs Student Reading Primary Grades Y 1 (All 
cohorts pooled, exploratory) 
All cohorts 
Control 
Level of random assignment 


Eaaarrees 
Y 1 ECTs in districts without formal mentoring programs 
with students in grades 4-6 eligible for state reading 15 12 
assessment 
5 


Y 1 ECTs in districts without formal mentoring programs 


with students in grades 4-6 eligible for state reading 0 
assessment who dropped during Y1 
Y1 ECTs in districts without formal mentoring programs [=e 
with students in grades 4-6 eligible for state reading 1 12 
assessment on rosters collected in spring 
Attrition 0% 
Differential attrition 0% 
Overall attrition 0% 
Student-level SSS 
Students in districts without formal mentoring programs 272 262 


on Y1, October 1 roster 

Students in districts without formal mentoring programs 
on Y1, October 1 roster with state reading assessment 267 253 
data next spring 

Students in districts without formal mentoring programs 


missing covariates 15 
Estimation Sample 238 
Attrition 9.2% 
Differential Attrition 1.5% 
Overall Attrition 8.4% 
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